Although most of the development tools from Openpilot are designed to interact with data on the cloud server from comma.ai, they also have the capability to work with data locally. However, the documentation for these tools may not make it clear how to use them in this way.
Fortunately, your device stores all of your Openpilot data logs on its own storage, so you can use tools like tools/replay or tools/plotjuggler to work with your local data instead of uploading and downloading data from connect.comma.ai. This can save a lot of time and effort.
This post provides a guide on how to access and work with local data using tools/replay and tools/plotjuggler.
0. Grabbing data logs from your device
To follow the steps in this guide, you will need to have SSH, ssh-keys, and connect your device to your hotspot or Wi-Fi. If you need help setting up SSH, please refer to the openpilot docs/SSH. We will assume that you have already set up SSH correctly for the rest of this guide.
A. By scp
a. Get all data from your device:
|
1 |
$ scp –r comma@<your–device–ip–or–hostname>:/data/media/0/realdata/* . |
b. Get specific segment from your device
|
1 |
$ scp –r comma@<your–device–ip–or–hostname>:/data/media/0/realdata/<segment–name> . |
For example, if I want to get segment 42 of 2022-12-11--19-22-46 route, then the command should be:
|
1 |
$ scp –r comma@<your–device–ip–or–hostname>:/data/media/0/realdata/2022–12–11—19–22–46—42 . |
B. By rsync
If you want to obtain rlog files without any videos and maintain the directory structure, using scp can be challenging. However, you can use rsync to achieve this easily.
For example, the following command can be used to obtain all rlog files and maintain the directory structure:
|
1 |
$ rsync –avR comma@<your–device–ip–or–host>:/data/media/0/realdata/2022–12–11—19–22–46*/rlog . |
This will give us something like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
$ tree data data └── media └── 0 └── realdata ├── 2022–12–11—19–22–46—0 │ └── rlog ├── 2022–12–11—19–22–46—1 │ └── rlog ├── 2022–12–11—19–22–46—10 │ └── rlog ├── 2022–12–11—19–22–46—11 │ └── rlog ├── 2022–12–11—19–22–46—12 │ └── rlog ├── 2022–12–11—19–22–46—13 │ └── rlog ├── 2022–12–11—19–22–46— ... |
1. Plotjuggler
Plotjuggler originally only supported cloud data, but added support for local data with the ntegan1 PR at May 9, 2022.
Assume we had the data directory above. First we will need to change the directory name to match how plotjuggler parse it by:
|
1 2 3 4 5 |
$ cd data/media/0/realdata; \ # Jump to directory of our routes for f in $(ls); do \ # Loop through all directories, you can filter here mv $f “ffffffffffffffff|$f”; \ # Change the directory name to match plotjuggler done; \ cd – # Jump back to where we from |
Then we can make plotjuggler to read and parse from our local data:
|
1 2 3 |
$ tools/plotjuggler/juggle.py \ —can \ “`pwd`/data/media/0/realdata/ffffffffffffffff|2022-12-11–19-22-46” |
If we want to change the directory name back, we can run this:
|
1 2 3 4 5 |
$ cd data/media/0/realdata; \ # Jump to data directory for f in $(ls | grep –E ‘f{16}\|’); do \ # Filter out all directories start with fffff… mv $f $(sed –E ‘s/^f{16}\|(.*)/\1/g’ <<<$f); \ # Change the directory name back done; \ cd – # Jump back to where we come from |
2. Replay
Similarly, tools/replay also only support cloud data before deanlee’s PR at Sep 20, 2021.
Assume we had the data directory above. We can use --data_dir option to replay data locally:
|
1 2 3 |
$ tools/replay/replay \ —data_dir data/media/0/realdata \ “ffffffffffffffff|2022-12-11–19-22-46” |
Note that, even though the directory does not contain dongle id, you still need to provide it in your command line argument (it could be fake as any 16 digit hex).
If you don’t want to – or you don’t have videos in your directory, you will need to pass --no-vipc to tell replay not to output videos.
3. Qt Cabana
Qt Cabana also support --data_dir option for local data logs, and it support from day one thanks to Dean Lee!
Assume we had the data directory above. We can use --data_dir option to replay data locally:
|
1 2 3 |
$ tools/cabana/cabana \ —data_dir data/media/0/realdata \ “ffffffffffffffff|2022-12-11–19-22-46” |
Note that Qt Cabana required at least road cam or qcam exist (we reuse loader from replay, but not allowing REPLAY_FLAG_NO_VIPC flag), otherwise it will shows the following error message:
|
1 |
no valid segments in route “ffffffffffffffff|2022-12-11–19-22-46” |
Leave a Reply