Bug #10276
closedCorrupt Radio Reference Importing Due to Not Properly Sanitizing Data
Added by C H almost 2 years ago. Updated almost 2 years ago.
100%
Description
Specific symbol I am having issues with is
’
Duplicate error by
attempting to import zip 27410
copy all data
go to file new
paste all data overwriting channel 0
export csv and save
now attempt to open saved file and you receive an error "unknown file format"
the problem is with field 90
GSO1 V.R.E.T.’S
To make the file load you have to delete the symbol it takes a long time to manually find problems and sanitize files with this error
Also another "bug" is when you import radio reference data the comments column does not show up unless you copy/paste the data to a new file
I also with comments data would be present alongside radio img files but it is not
Files
QueryResult.csv (12.3 KB) QueryResult.csv | C H, 01/16/2023 02:40 AM | ||
chirp_debug-uqj5qarh.txt (36.8 KB) chirp_debug-uqj5qarh.txt | C H, 01/16/2023 02:40 AM |
Updated by C H almost 2 years ago
C H wrote:
Specific symbol I am having issues with is
’
Duplicate error by
attempting to import zip 27410
copy all data
go to file new
paste all data overwriting channel 0
export csv and save
now attempt to open saved file and you receive an error "unknown file format"the problem is with field 90
GSO1 V.R.E.T.’S
To make the file load you have to delete the symbol it takes a long time to manually find problems and sanitize files with this error
Also another "bug" is when you import radio reference data the comments column does not show up unless you copy/paste the data to a new file
I also with comments data would be present alongside radio img files but it is not
ETA: problem is field 90 comment section
Updated by Dan Smith almost 2 years ago
- Target version set to chirp-py3
I cannot reproduce the copy/paste problem, all the channels paste cleanly for me, including the channel you mention. Can you please attach a debug log? See How_To_Report_Issues.
The missing comment field is a problem, I will fix that. Thanks!
Updated by C H almost 2 years ago
Platform is Windows 10
When I export my QueryResults to csv then attempt to open them it says "Unknown File Format".
Everything looks like it worked fine, the problem is when you try to OPEN the file.
The symbol above being present in the comments field is what is creating the issue. When I remove the symbol it works.
[2023-01-15 20:34:54,171] chirp.drivers.puxing_px888k - DEBUG: The file size does not match.
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.wxui.common - ERROR: <function ChirpMain.open_file at 0x00000237740B1360>: Unknown file format
Updated by C H almost 2 years ago
I went back further in debug log
`
[2023-01-15 20:34:23,618] chirp.wxui.query_sources - INFO: Starting QueryThread for
[2023-01-15 20:34:31,924] chirp.sources.radioreference - DEBUG: Mode 5 (Telm) is unsupported
[2023-01-15 20:34:54,171] chirp.chirp_common - DEBUG: Image data has no metadata blob
[2023-01-15 20:34:54,171] chirp.drivers.boblov_x3plus - DEBUG: Boblov_x3plus: match_model: size mismatch
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Drivers's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: ['R', '\x16\x90']
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Radio's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Drivers's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: ['P', '!@']
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Radio's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Drivers's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: ['T', 'G\x90']
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Radio's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Drivers's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: ['\', 'E\x10']
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Radio's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: 000: 4c 6f 63 61 Loca....
[2023-01-15 20:34:54,171] chirp.drivers.puxing_px888k - DEBUG: The file size does not match.
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.wxui.common - ERROR:
`
Updated by Dan Smith almost 2 years ago
- Status changed from New to In Progress
- Assignee set to Dan Smith
I see now, I thought you were saying it failed during copy/paste, I understand now, thanks.
Updated by Dan Smith almost 2 years ago
As I asked, please attach the full debug.log. Also attach the CSV file that you save out which fails to re-open. I need to see both attached to this bug.
I am still unable to reproduce this, even copy, paste, save as CSV and then re-open.
Updated by C H almost 2 years ago
- File chirp_debug-uqj5qarh.txt chirp_debug-uqj5qarh.txt added
- File QueryResult.csv QueryResult.csv added
Updated by Dan Smith almost 2 years ago
Okay, I can reproduce this, but only on Windows, not other platforms. This is a unicode encoding problem, not sanitization of special characters. I'll work on this.
Updated by C H almost 2 years ago
Yeah I actually noticed the characters were properly sanitized of the symbols when I opened the csv file with a different program right before sending the logs. Looking forward to the comments fix. Is there a way to have comments on your img files also? And another good feature would be setting TX power on CSV files because copying a csv file to a radio img I have to manually set all of my TX power settings.
I primarily use Linux myself but I have been on a different machine than I would normally use, Windows always creates issues with things for some reason...
Updated by Dan Smith almost 2 years ago
Yeah I actually noticed the characters were properly sanitized of the symbols when I opened the csv file with a different program right before sending the logs. Looking forward to the comments fix.
The comments fix is queued for tomorrow, so you'll have it then.
Is there a way to have comments on your img files also?
Not currently, as the image file is a direct 1:1 memory map of the radio. So, if the radio doesn't support it, we don't in the image.
And another good feature would be setting TX power on CSV files because copying a csv file to a radio img I have to manually set all of my TX power settings.
It's been on the list for a while.
Updated by C H almost 2 years ago
Sounds good, I will be on the lookout and I appreciate the prompt response.
Updated by Dan Smith almost 2 years ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
Applied in changeset github|91b4d7f4ddef8a46975bea3cc82bff2d0f6550b3.
Updated by C H almost 2 years ago
I don't know if you check closed issues but its amazing you fixed the issue this fast and implemented the features I was requesting. Thank you for doing so.
I can confirm that the unicode bug is gone.
But when it comes to the new feature of the power level being present in a csv sheet the results aren't exactly as I imagined although perhaps it is according to your development model.
I imagined setting the legal limits of all of the frequencies in a csv and copying and pasting the list to different radios and the configs basically set themselves for power level.
Just an imaginary example for testing not any real legal limits
lets say uv-5r has
high 4w
low 1w
ch1 has max legal power level of 2w set in the csv
what i imagine is that when you copy ch1 into the radio config, since 4w is higher than 2w that means you would need to automatically drop to the low power to be legal but this is not what happens,
in testing what happens is when i copy a 2w channel it sets the power to high(which on the radio is 4w), in my opinion it should set the power to low(1w) since that radios setting on high(4w) will exceed 2w(limit set in csv)
also I think there should be a .5w setting because of the top of my head at least certain frs channels still has a .5w limit
anyways if this was not your vision for how it should work I understand but I just figured I would give some constructive feedback
Updated by C H almost 2 years ago
also i think vhf marine is 25w, there are a lot of random numbers that aren't in the list
Updated by Dan Smith almost 2 years ago
Some of what you describe seems like it should be working. Either way, let's please discuss the power level stuff in a new bug unrelated to this unicode issue. Just open one and either I can fix/tweak stuff or explain why it has to be.