This vignette is a working draft. There will be lots of changes as
rvdat gets fleshed out, so please check back often!
The most recent changes were made on 2024-11-15.
Getting started
First, as we should always do when beginning a rvdat
workflow, we tell R where our vdat.exe is located. Mine is
within an installed version of Fathom
Connect.
vdat_here("c:/program files/innovasea/fathom connect/vdat.exe")
#> ℹ vdat is located at c:/program files/innovasea/fathom connect/vdat.exeIf you don’t want to do this every time you re-open R, you can add
VDAT_EXE="PATH_TO_VDAT.EXE" to your .Renviron
file. rvdat will automatically look for it there when you
restart R. You can do this manually, but I find that the easiest way to
edit .Renviron is using usethis::edit_r_environ().
Next, let’s download a sample VRL file from the glatos
package using utils::download.file. I’ll be keeping
everything in a temporary directory just to keep tidy.
td <- file.path(
tempdir(),
"vignette"
)
dir.create(
td
)
download.file(
url = file.path(
"https://github.com/ocean-tracking-network/glatos/raw/dev/inst/extdata",
"detection_files_raw/VR2AR_546310_20190613_1.vrl"
),
destfile = file.path(
td,
"VR2AR_546310_20190613_1.vrl"
),
mode = "wb",
quiet = TRUE
)Check the file’s metadata
To take a quick glimpse of the VDAT file’s metadata, we can use
vdat_inspect. Note that this only works for VDAT files
(.vrl and .vdat), not any exported CSVs.
vdat_insepct converts this information to a data.frame if
you’d like to keep it around.
inspect_df <- vdat_inspect(
file.path(td, "VR2AR_546310_20190613_1.vrl")
)
#> ==============================================================================
#> VRL
#> ==============================================================================
#> File: VR2AR_546310_20190613_1.vrl
#> Original: VR2AR_546310_20190613_1.vrl
#> Container: VR2AR VRL file (com.vemco.file.vrl.0207.ff02.ff02/5.0.1)
#> Created: 2019-06-13T13:30:54
#> Data UUID: ffee6ee7-450a-db42-b54c-334b672fddf5
#> Rx Model: VR2AR-69
#> Rx Serial: 546310
#>
#> ==============================================================================
#> Device
#> ==============================================================================
#> Decoding Map: MAP-113
#> Blanking Interval: 260 ms
#>
#>
head(inspect_df)
#> variable value
#> 1 File VR2AR_546310_20190613_1.vrl
#> 2 Original VR2AR_546310_20190613_1.vrl
#> 3 Container VR2AR VRL file (com.vemco.file.vrl.0207.ff02.ff02/5.0.1)
#> 4 Created 2019-06-13T13:30:54
#> 5 Data UUID ffee6ee7-450a-db42-b54c-334b672fddf5
#> 6 Rx Model VR2AR-69
#> section
#> 1 VRL
#> 2 VRL
#> 3 VRL
#> 4 VRL
#> 5 VRL
#> 6 VRLConverting VDAT file to a CSV
Now that we have a VRL to play with and know something about it,
let’s convert it to a csv. It’s pretty quick with
vdat_to_csv.
vdat_to_csv(
file.path(td, "VR2AR_546310_20190613_1.vrl"),
outdir = td
)
#> 0.0% - Converting File
7.6% - Converting File
9.2% - Converting File
10.7% - Converting File
12.2% - Converting File
13.7% - Converting File
15.3% - Converting File
16.8% - Converting File
18.3% - Converting File
19.8% - Converting File
21.4% - Converting File
22.9% - Converting File
24.4% - Converting File
26.0% - Converting File
27.5% - Converting File
29.0% - Converting File
30.5% - Converting File
32.1% - Converting File
33.6% - Converting File
35.1% - Converting File
36.6% - Converting File
38.2% - Converting File
39.7% - Converting File
41.2% - Converting File
42.7% - Converting File
44.3% - Converting File
45.8% - Converting File
47.3% - Converting File
48.9% - Converting File
50.4% - Converting File
51.9% - Converting File
53.4% - Converting File
55.0% - Converting File
56.5% - Converting File
58.0% - Converting File
59.5% - Converting File
61.1% - Converting File
62.6% - Converting File
64.1% - Converting File
65.6% - Converting File
67.2% - Converting File
68.7% - Converting File
70.2% - Converting File
71.8% - Converting File
73.3% - Converting File
74.8% - Converting File
76.3% - Converting File
77.9% - Converting File
79.4% - Converting File
80.9% - Converting File
82.4% - Converting File
84.0% - Converting File
85.5% - Converting File
87.0% - Converting File
88.5% - Converting File
90.1% - Converting File
91.6% - Converting File
93.1% - Converting File
94.7% - Converting File
96.2% - Converting File
97.7% - Converting File
100.0% - Converting File
#> ✔ File converted:
#> C:\Users\darpa2\AppData\Local\Temp\RtmpusjJ5z/vignette/VR2AR_546310_20190613_1.vrl
#> ℹ File saved in:
#> C:\Users\darpa2\AppData\Local\Temp\RtmpusjJ5z/vignette/VR2AR_546310_20190613_1.csv
list.files(td, pattern = "VR2AR")
#> [1] "VR2AR_546310_20190613_1.csv" "VR2AR_546310_20190613_1.vrl"vdat is very respectful of your data in that it never
overwrites files that exist in the same directory. It’ll just tack on a
time stamp to the name of the newly-exported file.
vdat_to_csv(
file.path(td, "VR2AR_546310_20190613_1.vrl"),
outdir = td
)
#> 0.0% - Converting File
7.6% - Converting File
9.2% - Converting File
10.7% - Converting File
12.2% - Converting File
13.7% - Converting File
15.3% - Converting File
16.8% - Converting File
18.3% - Converting File
19.8% - Converting File
21.4% - Converting File
22.9% - Converting File
24.4% - Converting File
26.0% - Converting File
27.5% - Converting File
29.0% - Converting File
30.5% - Converting File
32.1% - Converting File
33.6% - Converting File
35.1% - Converting File
36.6% - Converting File
38.2% - Converting File
39.7% - Converting File
41.2% - Converting File
42.7% - Converting File
44.3% - Converting File
45.8% - Converting File
47.3% - Converting File
48.9% - Converting File
50.4% - Converting File
51.9% - Converting File
53.4% - Converting File
55.0% - Converting File
56.5% - Converting File
58.0% - Converting File
59.5% - Converting File
61.1% - Converting File
62.6% - Converting File
64.1% - Converting File
65.6% - Converting File
67.2% - Converting File
68.7% - Converting File
70.2% - Converting File
71.8% - Converting File
73.3% - Converting File
74.8% - Converting File
76.3% - Converting File
77.9% - Converting File
79.4% - Converting File
80.9% - Converting File
82.4% - Converting File
84.0% - Converting File
85.5% - Converting File
87.0% - Converting File
88.5% - Converting File
90.1% - Converting File
91.6% - Converting File
93.1% - Converting File
94.7% - Converting File
96.2% - Converting File
97.7% - Converting File
100.0% - Converting File
#> ✔ File converted:
#> C:\Users\darpa2\AppData\Local\Temp\RtmpusjJ5z/vignette/VR2AR_546310_20190613_1.vrl
#> ℹ File saved in:
#> C:\Users\darpa2\AppData\Local\Temp\RtmpusjJ5z/vignette/VR2AR_546310_20190613_1[20241115-12-53-03-xxxxxx].csv
list.files(td, pattern = "VR2AR")
#> [1] "VR2AR_546310_20190613_1.csv"
#> [2] "VR2AR_546310_20190613_1.vrl"
#> [3] "VR2AR_546310_20190613_1[20241115-12-53-03-400292].csv"Converting a VDAT file to a CSV with time correction
Adding the time_corrected = TRUE argument to
vdat_to_csv corrects for clock drift using
vdat’s internal algorithm:
vdat_to_csv(
file.path(td, "VR2AR_546310_20190613_1.vrl"),
outdir = td,
time_corrected = TRUE
)
#> 0.0% - Extracting clock data
7.6% - Extracting clock data
9.2% - Extracting clock data
10.7% - Extracting clock data
12.2% - Extracting clock data
13.7% - Extracting clock data
15.3% - Extracting clock data
16.8% - Extracting clock data
18.3% - Extracting clock data
19.8% - Extracting clock data
21.4% - Extracting clock data
22.9% - Extracting clock data
24.4% - Extracting clock data
26.0% - Extracting clock data
27.5% - Extracting clock data
29.0% - Extracting clock data
30.5% - Extracting clock data
32.1% - Extracting clock data
33.6% - Extracting clock data
35.1% - Extracting clock data
36.6% - Extracting clock data
38.2% - Extracting clock data
39.7% - Extracting clock data
41.2% - Extracting clock data
42.7% - Extracting clock data
44.3% - Extracting clock data
45.8% - Extracting clock data
47.3% - Extracting clock data
48.9% - Extracting clock data
50.4% - Extracting clock data
51.9% - Extracting clock data
53.4% - Extracting clock data
55.0% - Extracting clock data
56.5% - Extracting clock data
58.0% - Extracting clock data
59.5% - Extracting clock data
61.1% - Extracting clock data
62.6% - Extracting clock data
64.1% - Extracting clock data
65.6% - Extracting clock data
67.2% - Extracting clock data
68.7% - Extracting clock data
70.2% - Extracting clock data
71.8% - Extracting clock data
73.3% - Extracting clock data
74.8% - Extracting clock data
76.3% - Extracting clock data
77.9% - Extracting clock data
79.4% - Extracting clock data
80.9% - Extracting clock data
82.4% - Extracting clock data
84.0% - Extracting clock data
85.5% - Extracting clock data
87.0% - Extracting clock data
88.5% - Extracting clock data
90.1% - Extracting clock data
91.6% - Extracting clock data
93.1% - Extracting clock data
94.7% - Extracting clock data
96.2% - Extracting clock data
97.7% - Extracting clock data
100.0% - Extracting clock data
0.0% - Converting File
7.6% - Converting File
9.2% - Converting File
10.7% - Converting File
12.2% - Converting File
13.7% - Converting File
15.3% - Converting File
16.8% - Converting File
18.3% - Converting File
19.8% - Converting File
21.4% - Converting File
22.9% - Converting File
24.4% - Converting File
26.0% - Converting File
27.5% - Converting File
29.0% - Converting File
30.5% - Converting File
32.1% - Converting File
33.6% - Converting File
35.1% - Converting File
36.6% - Converting File
38.2% - Converting File
39.7% - Converting File
41.2% - Converting File
42.7% - Converting File
44.3% - Converting File
45.8% - Converting File
47.3% - Converting File
48.9% - Converting File
50.4% - Converting File
51.9% - Converting File
53.4% - Converting File
55.0% - Converting File
56.5% - Converting File
58.0% - Converting File
59.5% - Converting File
61.1% - Converting File
62.6% - Converting File
64.1% - Converting File
65.6% - Converting File
67.2% - Converting File
68.7% - Converting File
70.2% - Converting File
71.8% - Converting File
73.3% - Converting File
74.8% - Converting File
76.3% - Converting File
77.9% - Converting File
79.4% - Converting File
80.9% - Converting File
82.4% - Converting File
84.0% - Converting File
85.5% - Converting File
87.0% - Converting File
88.5% - Converting File
90.1% - Converting File
91.6% - Converting File
93.1% - Converting File
94.7% - Converting File
96.2% - Converting File
97.7% - Converting File
100.0% - Converting File
#> ✔ File converted:
#> C:\Users\darpa2\AppData\Local\Temp\RtmpusjJ5z/vignette/VR2AR_546310_20190613_1.vrl
#> ℹ File saved in:
#> C:\Users\darpa2\AppData\Local\Temp\RtmpusjJ5z/vignette/VR2AR_546310_20190613_1[20241115-12-53-03-xxxxxx].csvA new column is added with time corrected down to the microsecond.
corrected <- read.csv(
list.files(td, pattern = "VR2AR.*\\]\\.csv$", full.names = TRUE)[2],
header = FALSE
)
corrected[46, c(2:3)]
#> V2 V3
#> 46 2019-06-11 02:47:00 2019-06-11 02:47:00.000725Splitting a VDAT file into a folder of CSVs
As outlined in vignette('vdat-data-structure'), the flat
CSV exported via vdat_to_csv is a mush-together of all
available data types in the VDAT file. vdat.exe can split
these into a folder of files according to data type for you, accessible
via rvdat in a few ways:
-
vdat_to_folder("PATH_TO_FILE"), which is a convenience wrapper around… -
vdat_to_csv("PATH_TO_FILE", folder = TRUE), which is a convenience wrapper around… -
vdat_call(c('convert', '--format=csv.fathom.split', 'PATH_TO_FILE')).
vdat_to_folder(
file.path(td, "VR2AR_546310_20190613_1.vrl"),
outdir = td
)
#> 0.0% - Converting File
7.6% - Converting File
9.2% - Converting File
10.7% - Converting File
12.2% - Converting File
13.7% - Converting File
15.3% - Converting File
16.8% - Converting File
18.3% - Converting File
19.8% - Converting File
21.4% - Converting File
22.9% - Converting File
24.4% - Converting File
26.0% - Converting File
27.5% - Converting File
29.0% - Converting File
30.5% - Converting File
32.1% - Converting File
33.6% - Converting File
35.1% - Converting File
36.6% - Converting File
38.2% - Converting File
39.7% - Converting File
41.2% - Converting File
42.7% - Converting File
44.3% - Converting File
45.8% - Converting File
47.3% - Converting File
48.9% - Converting File
50.4% - Converting File
51.9% - Converting File
53.4% - Converting File
55.0% - Converting File
56.5% - Converting File
58.0% - Converting File
59.5% - Converting File
61.1% - Converting File
62.6% - Converting File
64.1% - Converting File
65.6% - Converting File
67.2% - Converting File
68.7% - Converting File
70.2% - Converting File
71.8% - Converting File
73.3% - Converting File
74.8% - Converting File
76.3% - Converting File
77.9% - Converting File
79.4% - Converting File
80.9% - Converting File
82.4% - Converting File
84.0% - Converting File
85.5% - Converting File
87.0% - Converting File
88.5% - Converting File
90.1% - Converting File
91.6% - Converting File
93.1% - Converting File
94.7% - Converting File
96.2% - Converting File
97.7% - Converting File
100.0% - Converting File
#> ✔ File converted:
#> C:\Users\darpa2\AppData\Local\Temp\RtmpusjJ5z/vignette/VR2AR_546310_20190613_1.vrl
#> ℹ Files saved in:
#> C:\Users\darpa2\AppData\Local\Temp\RtmpusjJ5z/vignette/VR2AR_546310_20190613_1.csv-fathom-split
list.dirs(td, full.names = F, recursive = F)
#> [1] "VR2AR_546310_20190613_1.csv-fathom-split"
list.files(
list.dirs(td, full.names = T, recursive = F)
)
#> [1] "ATTITUDE.csv" "BATTERY.csv"
#> [3] "CFG_CHANNEL.csv" "CFG_TRANSMITTER.csv"
#> [5] "CLOCK_REF.csv" "DATA_SOURCE_FILE.csv"
#> [7] "DEPTH.csv" "DET.csv"
#> [9] "DIAG.csv" "DIAG_FAST.csv"
#> [11] "EVENT_INIT.csv" "EVENT_OFFLOAD.csv"
#> [13] "HEALTH_VR2AR.csv" "TEMP.csv"Other file types
Though we just used a VR2AR’s VRL file here, this works
for multiple different receiver styles and file types. For example, a
HR-180 VDAT file:
download.file(
url = file.path(
"https://github.com/ocean-tracking-network/glatos/raw/dev/inst/extdata",
"detection_files_raw/HR2-180_461396_2021-04-20_173145.vdat"
),
destfile = file.path(
td,
"HR2-180_461396_2021-04-20_173145.vdat"
),
mode = "wb",
quiet = TRUE
)
hr_output <- vdat_inspect(
file.path(td,"HR2-180_461396_2021-04-20_173145.vdat")
)
#> ==============================================================================
#> VDAT
#> ==============================================================================
#> File: HR2-180_461396_2021-04-20_173145.vdat
#> Original: HR2-180 461396 2021-04-20 173145.vdat
#> Container: Vemco Data File (com.vemco.file.vdat/1.0.0)
#> Content: HR2 Receiver Data Pack (com.vemco.file.vrdp.vrhr2/1.0.0)
#> Created: 2021-04-20T17:31:45
#> File UUID: f1a18604-46ea-4985-b57c-24809d37ad46
#> Data UUID: 1a595886-99ea-4ee9-a996-2e4d92dc01c3
#> Size: 39370
#> Blocks: 17
#> Block Types:
#> VRDP Header (com.vemco.file.vrdp.header/1.1.0)
#> RxLog General Summary (com.vemco.protobuf.rxlog.summary.general/1.0.0)
#> HR2 Offload Header (com.vemco.device.vrhr2.offload.header/1.0.0)
#> HR2 Offload Footer (com.vemco.device.vrhr2.offload.footer/1.0.0)
#> HR2 Detection Log Block (com.vemco.device.vrhr2.log.detect/1.0.0)
#> HR2 Ping Log Block (com.vemco.device.vrhr2.log.ping/1.0.0)
#> Codecs:
#> zlib compression (com.vemco.file.vdat.codec.zlib/1.2.11)
#>
#> ==============================================================================
#> VRDP
#> ==============================================================================
#> Creator: fathom-2.5.0-19700101--release
#> Source: HR2-180 (0814)
#>
#> ==============================================================================
#> Offload
#> ==============================================================================
#> Firmware Version: 2.0.1
#> Receiver Time @ Init: 2021-04-20T17:20:05Z
#> Client Time @ Init: 2021-04-20T17:20:06Z
#> Client Time Zone @ Init: -04:00
#> Receiver Time @ Offload: 2021-04-20T21:31:45Z
#> Client Time @ Offload: 2021-04-20T21:31:45Z
#> Client Time Zone @ Offload: -04:00
#>
#>
head(hr_output)
#> variable value
#> 1 File HR2-180_461396_2021-04-20_173145.vdat
#> 2 Original HR2-180 461396 2021-04-20 173145.vdat
#> 3 Container Vemco Data File (com.vemco.file.vdat/1.0.0)
#> 4 Content HR2 Receiver Data Pack (com.vemco.file.vrdp.vrhr2/1.0.0)
#> 5 Created 2021-04-20T17:31:45
#> 6 File UUID f1a18604-46ea-4985-b57c-24809d37ad46
#> section
#> 1 VDAT
#> 2 VDAT
#> 3 VDAT
#> 4 VDAT
#> 5 VDAT
#> 6 VDAT