Project

General

Profile

Actions

Bug #10241

open

Copy and Paste issues on Linux in a minimal environment

Added by Dennis Boone about 1 year ago. Updated about 1 year ago.

Status:
Feedback
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
01/08/2023
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Chirp Version:
next (py3)
Model affected:
Yaesu VX-7R
Platform:
Linux
Debug Log:
I read the instructions above:

Description

Various documentation indicates that I should be able to read my radio, then import from CSV to replace part or all of the defined channels. I can find no "Import" anywhere in the interface.

CHIRP next-20230108 on Linux ozymandias 5.19.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.19.6-1 (2022-09-01) x86_64 (Python 3.10.7)


Files

log (5.66 KB) log Debug log Dennis Boone, 01/08/2023 08:12 PM
2023-01-08-151522_1920x1200_scrot.png (165 KB) 2023-01-08-151522_1920x1200_scrot.png Screenshot of main screen Dennis Boone, 01/08/2023 08:17 PM
vx7r.csv (32.1 KB) vx7r.csv CSV file to overlay on radio image. Dennis Boone, 01/08/2023 08:50 PM
drb_vx7r_pre20230108.img (16 KB) drb_vx7r_pre20230108.img Radio image Dennis Boone, 01/08/2023 08:51 PM
log (5.99 KB) log Debug log Dennis Boone, 01/08/2023 08:51 PM
Screen Shot 2023-01-08 at 13.19.30.png (738 KB) Screen Shot 2023-01-08 at 13.19.30.png Dan Smith, 01/08/2023 09:20 PM
stderr (1.93 KB) stderr stderr messages Dennis Boone, 01/08/2023 09:26 PM
2023-01-08-162725_1920x1200_scrot.png (186 KB) 2023-01-08-162725_1920x1200_scrot.png CSV panel screenshot Dennis Boone, 01/08/2023 09:28 PM
2023-01-08-162745_1920x1200_scrot.png (170 KB) 2023-01-08-162745_1920x1200_scrot.png IMG panel screenshot Dennis Boone, 01/08/2023 09:28 PM
Screen Shot 2023-01-08 at 13.38.28.png (233 KB) Screen Shot 2023-01-08 at 13.38.28.png Dan Smith, 01/08/2023 09:39 PM
vx7r.csv (32.1 KB) vx7r.csv Updated fixed CSV Dennis Boone, 01/08/2023 10:01 PM
img.png (41.1 KB) img.png Dennis Boone, 01/08/2023 10:03 PM
properties.png (67 KB) properties.png Properties dialog after paste Dennis Boone, 01/08/2023 10:20 PM
log (4.84 KB) log Debug log of 3.11 unable to open a CSV Dennis Boone, 01/08/2023 11:31 PM
log (4.23 KB) log Log of next-20230111 Dennis Boone, 01/11/2023 10:49 PM

Subtasks 2 (0 open2 closed)

Bug #10244: Radios without ctone hide the tone value in wxUIClosedDan Smith01/08/2023

Actions
Bug #10245: Python 3.11 drops universal newline modeClosedDan Smith01/08/2023

Actions
Actions #1

Updated by Dan Smith about 1 year ago

  • Status changed from New to Feedback

The import dialog is part of the legacy interface and is no longer present in CHIRP-next. The suggested workflow is to just open the CSV file directly and copy/paste between them, which is more flexible and less confusing for most people.

Can you point me at the doc that needs updating?

Actions #2

Updated by Dennis Boone about 1 year ago

These places specifically say something about an Import function, or mention 'File > Import':

https://chirp.danplanet.com/projects/chirp/wiki/Beginners_Guide in the last section

https://chirp.danplanet.com/projects/chirp/wiki/CSV_HowTo in the Clean Start and Getting The Data Back Into A Radio method #1 sections

https://chirp.danplanet.com/projects/chirp/wiki/FAQ in the Transferring data between radios - I have two different radios, how do I transfer data between them section

There are several other places that just say 'import' without implying how.

BTW, thanks to you and contributors for python3 support. I was pretty much unable to run Chirp any more in Debian due to python2 going away.

Actions #3

Updated by Dennis Boone about 1 year ago

I'm having trouble with the paste operation. I've got an updated CSV file that I intend to use to completely replace the contents of the radio. It contains about 430 lines of memory data. Should I be able to block copypasta that?

Actions #4

Updated by Dan Smith about 1 year ago

Yep, you should be able to paste that. What trouble? Attach the debug log or the console output if there's anything obvious. You could also attach your CSV file here and I can try to repro.

Actions #5

Updated by Dennis Boone about 1 year ago

It seems to replace parts of line 1 (prompts to overwrite it), and nothing else.

Actions #6

Updated by Dan Smith about 1 year ago

Okay, I was able to paste the memories from your csv into your img, with some exceptions. Your CSV file has some things like a Cross mode of tone/tone which the vx7 doesn't support, so you can't paste that in. Like memory #15. That is cross tone/tone but with the same 100.0Hz tone. That should be TSQL with ctone of 100.0. When I paste them all, I get a bunch of warnings memories are incompatible with the vx7, for reasons like that tone issue, and some others where you have 2.5kHz steps. You don't see those warnings?

I don't see any errors in your log related to the paste-ing, so I would expect it should have pastes most of the memories, except for the ones it can't.

Actions #7

Updated by Dennis Boone about 1 year ago

Ok, Cross vs TSQL I think I can trivially fix. (I'm converting a local format into Chirp-edible CSV.)

I'm not seeing errors about that during the copy/paste run, though. I do see some errors about invalid power levels, which are in the debug log too.

Actions #8

Updated by Dan Smith about 1 year ago

Weird, this is the error I see when I copy 15 from the csv and paste it into the vx7. You don't see that? Maybe there's a linux thing going on.

Actions #9

Updated by Dennis Boone about 1 year ago

Definitely no dialog boxes like yours. Using an updated CSV that doesn't set 2.5 deviation, and uses TSQL instead of Cross, messages from stderr attached. Also two screen shots: of the csv with #15 selected, and the img with the corresponding row selected and the "will overwrite" dialog box.

Actions #10

Updated by Dan Smith about 1 year ago

Are the two screenshots showing that you can paste that memory now with the TSQL fix? I'll try to repro the missing error boxes on linux...

Actions #11

Updated by Dan Smith about 1 year ago

Weird, it works fine for me on Linux (ubuntu 20.04, distro packages for everything).

Actions #12

Updated by Dennis Boone about 1 year ago

Even as a single line copy-paste, it still ends up not really pasting everything. The name and tone value are blank.

Actions #13

Updated by Dennis Boone about 1 year ago

Is the 'missing locale dir' message from the debug log at all relevant here?

Actions #14

Updated by Dan Smith about 1 year ago

Even as a single line copy-paste, it still ends up not really pasting everything. The name and tone value are blank.

Are you saying that's the case for any copy/paste or just one/some specific channels?

I can do this procedure:

  1. Open both your .img and your .csv
  2. Edit memory 15 in your CSV to be TSQL/100.0
  3. Copy that line
  4. Switch to the .img, select memory 15, paste
  5. I get the "Overwrite?" dialog, click Yes
  6. Observe that memory 15 in the VX-7 img file has the same name (Dmndl), frequency (442.050)

I'm just noticing that the VX-7 driver isn't exposing has_ctone so that column is missing, which is maybe part of the confusion? But I definitely see the name...

No, the missing locale dir message is not relevant.

Actions #15

Updated by Dennis Boone about 1 year ago

The behavior seems the same whether I select one record or a large block. Select, copy, select, paste, I get the overwrite dialog and an incomplete copy. Results in the img file attached. (Note that I pasted into 11, the corresponding entry, rather than 15, but I don't think that should matter.)

Also attaching the updated CSV file that doesn't have the Cross/2.5 problems.

Actions #16

Updated by Dan Smith about 1 year ago

But are you copying 15 from the source file? I'm trying to narrow down the inputs here so I need to make sure we're looking at the same thing.

The "missing" tone thing is definitely a bug, but only in the display. I think that after you do the paste if you right click->properties, you'll see the tone is set. For that matter, can you check on the name as well?

I can't think of any reason the name wouldn't copy, especially since it does fine for me.

Just FYI, the paste operation is doing exactly the same code that import would loop through before - that part of the code hasn't changed in a looong time.

Actions #17

Updated by Dennis Boone about 1 year ago

Sorry, I keep leaving bits out of my description.

If I do this as a single line copy-paste:

Select 15 in revised csv. Copy. Select 11 in img. Paste. Get overwrite dialog, click yes. Results: tone and name are missing or wrong. Tmode and tuning step change to the values from the CSV. Power changes to L1.

If I do this as a block copy-paste:

Select row 1 in CSV (CSV comes in with a "0" record that's blank, for whatever reason). Shift-page/cursor down to include everything through 437. Copy. Select row 1 in img. Shift-page/cursor down to include everything through 437. Paste. Get overwrite dialog, click yes. Results: in row 1, missing tone and name, power changed to L1, no other entries appear to have been touched, only 255 entries exist.

Properties dialog screenshot from img file after single-row paste is attached.

Actions #18

Updated by Dan Smith about 1 year ago

Select 15 in revised csv. Copy. Select 11 in img. Paste. Get overwrite dialog, click yes. Results: tone and name are missing or wrong. Tmode and tuning step change to the values from the CSV. Power changes to L1.

Okay, the tone thing will be fixed as part of #10244 . I totally cannot reproduce your missing name thing, and from the post-paste properties screenshot, it looks like the tone isn't updated either, which I also cannot reproduce.

If I do this as a block copy-paste:

Select row 1 in CSV (CSV comes in with a "0" record that's blank, for whatever reason).

Because CSV files' memories start at zero like many other non-Yaesu radios.

Shift-page/cursor down to include everything through 437. Copy. Select row 1 in img. Shift-page/cursor down to include everything through 437. Paste. Get overwrite dialog, click yes. Results: in row 1, missing tone and name, power changed to L1, no other entries appear to have been touched, only 255 entries exist.

I wonder if whatever is preventing you from getting the dialog about the unsupported memories is related here. The partial single-line paste, and the only-trying-the-first-row thing ... maybe it's choking on displaying that box and not going further or something. I see you're running against wxPython 4.2.0, and you installed from the wheel right? Where is your wxpython coming from? Do you have python3-wxgtk* installed via debian packages, or something from pip?

Actions #19

Updated by Dennis Boone about 1 year ago

Looking at the METADATA file in the wheel, there's:

Requires-Dist: importlib-resources ; python_version < "3.10"

but python is:

Python 3.10.7 (main, Sep 8 2022, 14:34:29) [GCC 12.2.0] on linux

Also, I don't seem to have importlib-resources installed. Any of this relevant here?

Actions #20

Updated by Dan Smith about 1 year ago

Dennis Boone wrote in #note-19:

Looking at the METADATA file in the wheel, there's:

Requires-Dist: importlib-resources ; python_version < "3.10"

but python is:

Python 3.10.7 (main, Sep 8 2022, 14:34:29) [GCC 12.2.0] on linux

Also, I don't seem to have importlib-resources installed. Any of this relevant here?

No, importlib-resources is required for pythons older than 3.10. You have 3.10 so you don't need (or want) it installed.

Actions #21

Updated by Dennis Boone about 1 year ago

Versions of stuff from pip:

requests 2.27.1
six 1.16.0
future 0.18.2
yattag 1.15.0
wxPython 4.2.0

I downloaded the .whl file earlier today, and installed it with pip install chirp-next_20230108-py3-none-any.whl. wxPython seems to have come in via pip. pyserial is 3.5, and came in via apt. Python3 itself came in via apt also. python3-wxgtk4.0 via apt.

Actions #22

Updated by Dan Smith about 1 year ago

Can you uninstall wxpython via pip? What's the version of python3-wxgtk4.0 in apt? Hoping it's 4.0.7ish.

Actions #23

Updated by Dennis Boone about 1 year ago

Hmm, not sure what's going on with the pip-listed wxPython. When I tried to have pip remove it, it grumbled about it being outside /usr.

Here's the apt line for python3-wxgtk4.0 -- newer than 4.0.7:
ii python3-wxgtk4.0 4.2.0+dfsg-1 amd64 Python 3 interface to the wxWidgets Cross-platform C++ GUI toolkit

Actions #24

Updated by Dennis Boone about 1 year ago

It might be useful to note here that this box runs Debian Sid.

Actions #25

Updated by Dennis Boone about 1 year ago

Arrrrgh, in the process of removing and re-installing the apt-provided python3-wxgtk4.0 package, I had to remove a bunch of stuff with interlocking dependencies, and when I reinstalled the ones I cared about, I got a python3 upgrade to 3.11.1.

Actions #26

Updated by Dan Smith about 1 year ago

Okay, but does that mean you got the pip-installed wxpython uninstalled or not?

You can tell where your wx is coming from with:

$ python
import wx
print(wx.__file__)

We run 4.2.0 on macos and Windows, so I would expect that's fine. However, both of my linux test boxes provide 4.0.7. Just looking for differences that might explain how or why you'd not be getting the incompatible dialog box, if that's even the problem. Running with the distro package for wxpython would definitely be my suggestion for minimal problems.

I've done no testing on python 3.11, although I know it has broken things for other projects .. so not sure if that will help or hurt.

Actions #27

Updated by Dennis Boone about 1 year ago

ozymandias 62 # python
Python 3.11.1 (main, Dec 31 2022, 10:23:59) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import wx
>>> print(wx.__file__)
/usr/lib/python3/dist-packages/wx/__init__.py
>>> 
ozymandias 63 # dpkg -L python3-wxgtk4.0
/.
/usr
/usr/lib
/usr/lib/python3
/usr/lib/python3/dist-packages
/usr/lib/python3/dist-packages/wx
/usr/lib/python3/dist-packages/wx/__init__.py
/usr/lib/python3/dist-packages/wx/__version__.py
/usr/lib/python3/dist-packages/wx/_adv.cpython-310-x86_64-linux-gnu.so
...

I'm now getting an "unknown file format" dialog trying to open the CSV, so I'll have to explicitly use 3.10 for now. All of the package wrangling hasn't changed the behavior, though.

Actions #28

Updated by Dennis Boone about 1 year ago

I think pip is seeing the apt-installed stuff, partially.

ozymandias 306 # pip show -f wxPython
Name: wxPython
Version: 4.2.0
Summary: Cross platform GUI toolkit for Python, "Phoenix" version
Home-page: http://wxPython.org/
Author: Robin Dunn
Author-email: robin@alldunn.com
License: wxWindows Library License (https://opensource.org/licenses/wxwindows.php)
Location: /usr/lib/python3/dist-packages
Requires: 
Required-by: 
Files:
Cannot locate RECORD or installed-files.txt
Actions #29

Updated by Dan Smith about 1 year ago

I'm now getting an "unknown file format" dialog trying to open the CSV, so I'll have to explicitly use 3.10 for now.

Um, that would be extremely surprising if that were a 3.11 thing...

All of the package wrangling hasn't changed the behavior, though.

Okay, well, I'm not sure what else to try since I can't reproduce any of that locally. I've queued an extra log statement for tomorrow's build that you could try to patch in if you want:

https://github.com/kk7ds/chirp/pull/367/commits/400ec735a9c82b25094543348da660ebbffcbd14

Which will expose the paste errors to the log (stderr) so we can see them even if the dialog isn't being shown. However, reviewing the code in the paste routine there, it will try to paste everything and then just report the errors in one step, so that pretty much shoots my idea that failure to show the dialog was stopping it short. Certainly the half paste of a single row doesn't follow that reasoning anyway, since the underlying operation is an atomic set_memory() to the driver, which the UI repopulates from after refresh.

Actions #30

Updated by Dennis Boone about 1 year ago

Here's the debug log for a 3.11 run that experienced the CSV format error thing.

Actions #31

Updated by Dennis Boone about 1 year ago

I'll try the extra logging tomorrow. If you have any other suggestions as to tracing/debugging strategy, I can dig into them as well.

Actions #32

Updated by Dan Smith about 1 year ago

[2023-01-08 18:27:30,668] chirp.directory - ERROR: Radio class CSVRadio failed during detection: invalid mode: 'rU'

Looks like more "screw compatibility, break things in every version" from the python team :(

Actions #33

Updated by Dennis Boone about 1 year ago

I just tried CHIRP next-20230111. The weird CSV reading problem is gone. But the copypasta issue still exists.

Actions #34

Updated by Dennis Boone about 1 year ago

While I still don't know what is really going on here, after various CSV file changes, I did get Chirp to put up a dialog saying several hundred memories would be replaced, and the results seem to be rational. Maybe we walk away while the walking is good? :) Thanks for your assistance!

Actions #35

Updated by Dan Smith about 1 year ago

For what it's worth, I've spent hours trying to reproduce your results and have failed.

I was going to suggest that you try copy and paste between csv tabs to narrow down the problem to something VX7-driver-specific or a generic GUI problem.

I figure I'll have to wait until someone else reports a similar problem so I have another data point, because I'm pretty much out of ideas, sorry :(

Actions #36

Updated by Dan Smith about 1 year ago

I just had an idea. What desktop environment are you running? When you copy and paste, chirp sends two things to the clipboard: a rich data object for chirp-to-chirp copy/paste, and a text format for if you paste into another application. I wonder if your desktop environment's clipboard is only persisting the plaintext version?

Actions #37

Updated by Dennis Boone about 1 year ago

I just run a window manager, not a full blown gnome or kde sort of thing. I do have xclipboard running. I'll have to experiment with this a bit to see if I can cause the behavior to change.

Actions #38

Updated by Dan Smith about 1 year ago

  • Subject changed from Import function missing to Copy and Paste issues on Linux in a minimal environment

Aha! I bet this is it. The text representation is not the full deal, so that would totally explain the difference. If you could start a full desktop instance just to test, that might be worth it.

I've been meaning to, but I'm going to change the title of this bug to more accurately represent where it has gone, if that's okay, since I think I scrubbed the import from the docs early on :)

Actions #39

Updated by Dennis Boone about 1 year ago

Yes, title change makes sense. I probably should have opened a second issue rather than just forging ahead.

I spent a few minutes playing with an alternate clipboard manager which will show me the content types, and I'm definitely getting just text/plain. Interestingly, the stuff which winds up in the clipboard is different for Ctrl-C ([146.010000/+0.600/67.0/69.3]) vs Edit menu/Copy (hundreds of lines, including all the empty rows, with sort of a columnar text version of the content items).

It'd take some work to spin up a full gnome desktop environment, but I'll do some more fiddling here.

What type is Chirp trying to use with the clipboard?

Actions #40

Updated by Dan Smith about 1 year ago

I spent a few minutes playing with an alternate clipboard manager which will show me the content types, and I'm definitely getting just text/plain. Interestingly, the stuff which winds up in the clipboard is different for Ctrl-C ([146.010000/+0.600/67.0/69.3]) vs Edit menu/Copy (hundreds of lines, including all the empty rows, with sort of a columnar text version of the content items).

Edit/Copy should (and does for me) only copy the selected rows. Perhaps there's a situation where you're selecting everything or have selected nothing and it grabs the whole sheet? I'll have to poke around there.

It'd take some work to spin up a full gnome desktop environment, but I'll do some more fiddling here.

Cool, well, sounds like we're onto something here then.

What type is Chirp trying to use with the clipboard?

Well, it's defined through wx, but I give it a mimetype of "x-chirp/memory-channel" but I dunno if that actually makes it to the clipboard like that.

Actions #41

Updated by Dennis Boone about 1 year ago

Edit/Copy should (and does for me) only copy the selected rows. Perhaps there's a situation where you're selecting everything or have selected nothing and it grabs the whole sheet? I'll have to poke around there.

I had done select-all (using the top left corner of the grid).

Well, it's defined through wx, but I give it a mimetype of "x-chirp/memory-channel" but I dunno if that actually makes it to the clipboard like that.

I wonder if it's about the type. I'll have to see if I can come up with a way to frob that a little.

Actions

Also available in: Atom PDF