Anticipated Nov 2009 Update of RDF/Map/Image Files

[gfb107]

Use of the button code in the map also allows for the renaming of buttons for different devices and or modes, or even between remotes that share a common image.

The discussion about that is here.

[WagonMaster]

Thanks for that link, Greg. It appeared to have a lot of the history, so I wanted to take the time to read the whole (6-year-old!) thread, but in retrospect, it didn't enlighten me as much as I'd hoped. But it was worth the read nevertheless.

Use of the button code in the map also allows for the renaming of buttons for different devices and or modes, or even between remotes that share a common image.

I understand the last part of that -- you might have 2 map files, each referencing the same image file, but using different button names. Without looking, I don't know if there are any current examples of that, but I appreciate the logic.

But I don't understand the "allows for the renaming of buttons for different devices and or modes" part. Can you expand on that, please? And maybe give an example to help me understand?

So, in my confused state, I'm still struggling with why we should have both the button code and the button name in the map file.

Having both of them potentially provides some theoretical remote-drawing, button-labeling application with the ability to avoid reading the RDF but at the real-world cost of adding the potential for a button name discrepancy.

So, the more I've thought about this, the more I think that there's a valid argument to use only button codes in the map files. In fact, you (Greg) suggested that very thing in the referenced thread and Nils seemed to agree that it was a good idea. It wasn't clear to me why that idea didn't persevere, though. It sounds more logical and less error-prone than what we have currently, since it would nicely remove the whole issue of conflicting names for buttons in the RDF and map files.

In fact, I'll take it a step further and go so far as to wonder aloud: Why shouldn't the data from the map files just be part of the RDF? That would really eliminate the problem. The details about the remote's button positions and sizes (in conjunction with the image filename, of course) seem like a logical candidate for the RDF anyway. And this would reduce the set of files needed to specify everything you need to know about a remote control from 3 files to just 2 (or even 1 if the image data was encoded/decoded directly into/from the RDF with 'uuencode'/'uudecode' or similar -- think about that for a second -- the RDF would then cover everything!).

I envision a "[Buttons]" section with entries like this, 1 button definition per line (example derived from the URC-10820's files):

Code: Select all

Play=$0c :: circle 70,103 89,111
"up arrow":Up=$31 :: poly 49,185 59,195 70,191 81,195 90,185 79,178 69,176 58,179 49,185 49,185

Now maybe this suggestion is problematic for reasons I don't envision (beyond the obvious burden on the apps which read RDFs and the overhaul of the RDFs to incorporate that data, the latter of which could probably be automated). I'm certainly not suggesting we change anything in the short term, but I think it might be valuable to have the discussion.

What apps use the map files? I think only RM does. And since RM has to read the RDFs anyway, putting the map data into the RDF would cause no RDF-reading penalty to applications which only use the map file because there are none and probably never would be any, I suspect.

On a different but related note, from that thread, it almost sounded like, at one point in the past, RM would detect a mismatch in the button name used in the RDF and map files. Based on that, I tested it. Vicky's tool shows that the RS 15-100 remote has 19 cases where buttons are defined in the map file but not in the RDF. So I loaded up RM 1.98beta4 and selected the 15-100 remote. There's no warning or error message, but I noticed that, for example, the "Record" button isn't circled -- it's not available -- essentially just as predicted by Vicky's tool report. After shutdown of RM, I looked in 'rmaster.err' and sure enough, I see this:

Code: Select all

Searching for remote with name RS 15-100
Warning: Shape defined for unknown button SAT_CBL
Warning: Shape defined for unknown button AUDIO
Warning: Shape defined for unknown button pvr_menu
Warning: Shape defined for unknown button thumbs_down
Warning: Shape defined for unknown button tv/vcr
Warning: Shape defined for unknown button record
Warning: Shape defined for unknown button rewind
Warning: Shape defined for unknown button fast_forward
Warning: Shape defined for unknown button MOVE_skip-
Warning: Shape defined for unknown button PIP_live-tv
Warning: Shape defined for unknown button SWAP_skip+
Warning: Shape defined for unknown button thumbs_up
Warning: Shape defined for unknown button fav_scan
Warning: Shape defined for unknown button page_down
Warning: Shape defined for unknown button page_up
Warning: Shape defined for unknown button arrow_left
Warning: Shape defined for unknown button arrow_up
Warning: Shape defined for unknown button arrow_down
Warning: Shape defined for unknown button arrow_right

Shouldn't these errors be more prominent, maybe annunciated with a single pop-up window? It seems to me like RM would be a good place to flag this. Even though Vicky's tool catches these problems so that they can be fixed before release of the RDF/map files, having a secondary check would be a good thing, especially as it seems that we have erroneous map/RDF files out there now.

From a different angle, I'm surprised that over 600 "missing button" errors (across several variants of remotes) could have gone undetected for so long. It seems to me that someone would be using their remote, find out that there's a button they can't access and complain.

Confused... (in more ways than one! )

Bill

[mdavej]

I know your questions are for Greg, but I have some info which may shed some light.

My 15-100 (modified version of Rob's) has zero errors. Rob's version has zero errors. But if you took my RDF and his MAP (or vice versa), you'd get tons of errors, as you found. My version uses the same file names as Rob's but his are incompatible with mine. So I believe your analysis may be flawed as your tool may be comparing files which should not be compared. A map can only be compared with the the RDF version the map author used, not any other versions.

Why did I make my own version and muck up things? Personal preference mostly on the button names (I find all caps hard to read) and to tidy up the button shapes. I also know that there are maps, images and possibly even other versions of RDF's scattered all over, mostly in the diagnosis area. One of my favorite atlas maps I got from the diagnosis area, by xnappo, IIRC.

As for adding the map data to the RDF, although I like the idea of consolidating information into fewer files, I vote a resounding No. The map is built and maintained by a totally different, but very real remote-drawing, button-labeling application called "MapThis!". Whoever makes or edits a map would have to paste the results into the RDF, which would be tedious and error prone. And if there is no separate Map file, it will be impossible to ever edit it again with the MapThis application. Plus we have tighter control on RDFs, since most are created by a handful of experts. Maps, on the other hand are created by many other people with varying degrees of programming expertise, who may not be comfortable editing RDF's.

I'll leave the final call to Greg regarding names versus codes. But my 2 cents is that RM can already handle either or both, and there are so many map files out there we risk breaking, I don't think it's worth messing with. And as I said earlier, I don't think most the errors you've found programmatically are really there, or we would have certainly heard complaints, like you said.

I don't think the number of files is the problem. The problem is we have no single install package for all the required JP1 tools and associated files. If I had the skills, I'd make one myself. That's really where any new consolidation efforts should be focused, IMO.

[vickyg2003]

From a different angle, I'm surprised that over 600 "missing button" errors (across several variants of remotes) could have gone undetected for so long. It seems to me that someone would be using their remote, find out that there's a button they can't access and complain.

Missing buttons in Kameleon Remotes need to be ignored, since each screen has different buttons, and not all buttons are on all screens.

So, in my confused state, I'm still struggling with why we should have both the button code and the button name in the map file

One thing about having them both in the rdf and the map is it makes it easier to find if there are errors. Often times on first creation it is easy to make a mistake in the button Hex. Those errors are usually only discovered when an upgrade is updated sometimes weeks or months after the creation of the map and rdfs.

[gfb107]

The main reason for not putting the image map information in the RDF is that the map files are a standard format, and the tool we use to create/edit them will not read/write an RDF.

There are some situations where we expect that some of the shapes defined in a map file will not have corresponding buttons defined in the RDF. I don't remember an example right now.

It is also possible to have multiple shapes for the same (RDF) button. One example is the URC-6131n, which has two physical buttons (Audio and CC) that use the same button code. This is supported by using the button code to associate the shape with the RDF button definition, then having a different display name for each shape.

[WagonMaster]

First off, thanks to all who replied to my last post. Sorry for the delay in replying since then. I'll break my replies into logically separate posts since I've seen the forum crash mightily when it tries to digest a long post.

From a different angle, I'm surprised that over 600 "missing button" errors (across several variants of remotes) could have gone undetected for so long. It seems to me that someone would be using their remote, find out that there's a button they can't access and complain.

Missing buttons in Kameleon Remotes need to be ignored, since each screen has different buttons, and not all buttons are on all screens.

Excellent point. I actually knew that but just did a simple line count on your report and didn't think to filter out the "Kameleon" lines. So let me try again:

Code: Select all

==> grep "map" NoButtonRDF.txt | grep -vi "kameleon" | wc -l
511

OK, not 603 errors, only a mere 511 errors. I feel much better now.

All kidding aside, Dave and Vicky are certainly correct in suggesting that some of the errors are not "really there". But the real number is almost certainly not zero either and the only way to know what's a real error and what isn't is to work your way through them, hopefully eliminating them in large batches. The point here is that the job is made more difficult every time someone creates a new map file or RDF and doesn't push the changes "upstream" for inclusion in the master set. Now, in Dave's case, he's properly made the RDF/map file available on the forum, but Nils sent me something different from Dave's 15-100 files and (in the case of the RDF) different from Rob's 15-100 files. So I have 2 unique maps and 3 unique RDFs. If I want to be diligent (I do), I still have to work my way through them, comparing the permutations, and making sure nothing is amiss. However, I'm worried that if there are too many cases like this, I may not meet that self-imposed Nov 2009 deadline -- Dec 2009, here we come!

BTW, Dave, don't take it personally that I used the RS 15-100 as my example of these errors. That was just the first one that appears in Vicky's tool's report's (alphabetical) list! For your next RDF/map efforts, consider working on the 'ZapStation'.

Oh, and that's just one type of error -- there are others. Vicky knows -- she wrote the tool that's finding them! But even as I inevitably encounter many "false positives", I know the effort is worth it (on her part and my part).

I better not say any more or I won't be able to hand this job off to some other sucker, err, participant when the time comes.

Dave, speaking specifically about the 15-100 (since that's where I'm beginning), after examination of all the files, I'm planning to use your map file and a modified version of your RDF. But I see that your version of the RDF (from the forum) is missing the "[SetupCodes]" section whereas Nils' version (emailed to me) does have that section, so I'll be using it to populate your RDF. I notice that your RDF is missing the "SetupValidation=Enforce" line and it's in Nils' version. Was that intentional? Should it be in the final RDF?

There are also 3 variants of this part of the RDF buttons defs:

"slow-":Check=46,
"slow+":Uncheck=45,

"slow-":thumbs_down=45,
"slow+":thumbs_up=46,

"slow-":Uncheck=45,
"slow+":Check=46,

Pay special attention to the fact that Nils' variant has the code values swapped. Which of these 3 is correct??? My plan is to use yours (Dave's), but the fact that Nils' version is more up-to-date than Rob's in some ways makes me wonder if Nils' has the correct codes and you and Rob have the wrong codes or vice-versa.

As you can see, this is what makes the job harder than it should be. I wish people who author and/or tweak these files would coordinate their changes and figure out which version is best (or, as in this case, create the best one from a combination of each) and then delete the obsolete files from the forum so that nobody (especially the maintainer) gets confused. I'm confused enough as it is!

Bill

[WagonMaster]

One thing about having them both in the rdf and the map is it makes it easier to find if there are errors. Often times on first creation it is easy to make a mistake in the button Hex. Those errors are usually only discovered when an upgrade is updated sometimes weeks or months after the creation of the map and rdfs.

As for adding the map data to the RDF, although I like the idea of consolidating information into fewer files, I vote a resounding No. The map is built and maintained by a totally different, but very real remote-drawing, button-labeling application called "MapThis!". Whoever makes or edits a map would have to paste the results into the RDF, which would be tedious and error prone. And if there is no separate Map file, it will be impossible to ever edit it again with the MapThis application. Plus we have tighter control on RDFs, since most are created by a handful of experts. Maps, on the other hand are created by many other people with varying degrees of programming expertise, who may not be comfortable editing RDF's.

I feel the need to clarify something Dave said above about after-the-fact editing. "Impossible" is way too strong a word. "Inconvenient" is much more appropriate. I mean, since 'Map This!' (wisely) uses basic ASCII text files, it's not a 1-way street. The data can be moved in and out as needed. No, it's not as convenient as keeping the data in its original 'Map This!' file, but it's certainly not that hard or that frequent a task that anyone serious about tweaking it should have much of a complaint. And if someone was really doing this a lot, it wouldn't be hard to whip up a little script to move the data back and forth, in which case "inconvenient" becomes "mildly inconvenient".

Having said that, I still appreciate everyone's input. I didn't really expect a lot of enthusiasm for such a change but thought (and still think) that it's worth discussing even if it winds up changing nothing. The discussion might spur some other interesting ideas or improvements in other areas.

There are some situations where we expect that some of the shapes defined in a map file will not have corresponding buttons defined in the RDF. I don't remember an example right now.

Well, since Vicky's tool will detect that, I should get a good idea of what those cases are, once I filter through all 511 errors. I'm going to be very interested in finding out what those are and if they're legitimate or not.

Bill

[WagonMaster]

I'll leave the final call to Greg regarding names versus codes. But my 2 cents is that RM can already handle either or both, and there are so many map files out there we risk breaking, I don't think it's worth messing with.

You're probably right. But I suspect if we did ever want to change this, it might be worth automating, making it easier and (theoretically) safer.

It is also possible to have multiple shapes for the same (RDF) button. One example is the URC-6131n, which has two physical buttons (Audio and CC) that use the same button code. This is supported by using the button code to associate the shape with the RDF button definition, then having a different display name for each shape.

I'm having trouble understanding this, Greg. I have just 2 maps for the URC-6131 variants: 'URC-6131.map' and 'URC-6131nw.map'. So I assume you mean the 'URC-6131nw.map'. But when I look at that map, I see distinct physical buttons with unique button codes, on 2 adjacent lines:

Code: Select all

circle $37:Audio  70,382 78,390
circle $45:CC{Closed_Caption}  46,381 54,389

It's actually the 'URC-6131.map' file that has the shared button, it seems:

Code: Select all

circle $37:Audio{CC}  51,275 59,283

Nils sent me 3 unextended versions of a URC-6131 RDF. Two of the 3 list both map files on the "ImageMap=" line and the 3rd lists only the 'URC-6131nw.map' file.

PVR0PVR0 (URC-6131(Old)_6131nwB00 PVR Remote 1K).rdf:

Code: Select all

input:Audio{CC}=$37,

PVR0PVR0 (URC-6131(Old)_6131nwB00 PVR Remote 2K).rdf:

Code: Select all

input:Audio{CC}=$37,

PVR0PVR0 (URC-6131nwB00 PVR Remote 2K FIXED).rdf

Code: Select all

CC:ClosedCaption=$45,
input:Audio=$37,

So, as near as I can see, neither of the map files have "two physical buttons (Audio and CC) that use the same button code". Do I have the same map/RDF files you do? If so, can you please clarify what you meant? If not, please post yours somewhere so I can resolve this discrepancy.

Aside: There was a diversion as I investigated this that I want to explain. I ran 'Map This!' on the -6131 map files to try to better understand this whole discussion, specifically the point Greg was making.

The 'URC-6131.map' file looked fine but the 'URC-6131nw.map' had all these weird, large rectangles! To make a long story short, I realized that whoever made that file must've done some sort of hand-editing to the 'rect' lines. The 2 coordinate pairs were erroneously separated by a comma instead of by a space! Once I figured that out and fixed it, the large erroneous rectangles were all replaced by proper button-sized rectangles.

Oh, and I ran an automated check of all the map files for such corruption. That was the only file so affected -- of course... the one I happen to load today is broken!

Curious as to why this was never detected, I fired up RemoteMaster using the old, broken map file and found that it processed the rectangles correctly, which is technically an error since the rectangles were mis-encoded. Probably not anything to worry about, Greg, but I mention it in case you want to address that minor issue.

Bill

[vickyg2003]

Last time we had an update, Nils and kupakai took on the Herculean task of creating most of the maps as well as updating the RDF's to a new standard. And just like this time there were all sorts of versions out there, because of the time involved from start to finish.

I think that 1 thing that really contributes to the problem you are facing is the way we update RDFs and images.

If we EDIT instead of DELETE and ADD, we hide changes from the RDF Librarian. Therefore we get verisions in all states.

When you sort the file for last update, it really shows you when it was ADDED.

[mdavej]

Dave, speaking specifically about the 15-100 (since that's where I'm beginning), after examination of all the files, I'm planning to use your map file and a modified version of your RDF. But I see that your version of the RDF (from the forum) is missing the "[SetupCodes]" section whereas Nils' version (emailed to me) does have that section, so I'll be using it to populate your RDF. I notice that your RDF is missing the "SetupValidation=Enforce" line and it's in Nils' version. Was that intentional? Should it be in the final RDF?

There are also 3 variants of this part of the RDF buttons defs:

"slow-":Check=46,
"slow+":Uncheck=45,

"slow-":thumbs_down=45,
"slow+":thumbs_up=46,

"slow-":Uncheck=45,
"slow+":Check=46,

Pay special attention to the fact that Nils' variant has the code values swapped. Which of these 3 is correct??? My plan is to use yours (Dave's), but the fact that Nils' version is more up-to-date than Rob's in some ways makes me wonder if Nils' has the correct codes and you and Rob have the wrong codes or vice-versa.

I'm glad to see you going through these with a fine tooth comb. No offense taken, and I can see you're taking my comments constructively as well.

Here's the deal with the mismatch above. When I first released my RDF, I did indeed have those two keys backwards (see THIS thread). That's probably the version Nils started with, which is incorrect. He went on to add setup codes and validation to that version. I later fixed the swapped keys, so what you posted is correct. But I didn't update it any further afterward with setup codes or validation. So it's a good thing you're trying to reconcile these.

Bottom line, setup codes and validation should be in. Rob's and my check/uncheck keys are correct as posted. Since the actual buttons have check marks printed on them, I used that in my description. Thumbs up/down is also valid since that's common on other remotes for the same kinds of functions. Bill based his 15-100 extender RDF on mine, so his works with my map too. On the other hand, Rob created the RDF to begin with, and thumbs up/down are used in KM.

[WagonMaster]

I think that 1 thing that really contributes to the problem you are facing is the way we update RDFs and images.

If we EDIT instead of DELETE and ADD, we hide changes from the RDF Librarian. Therefore we get verisions in all states.

When you sort the file for last update, it really shows you when it was ADDED.

Now that's interesting. I don't think I noticed that before. I just tried it. If you sort by "Date", it works as expected and the "Date" column is as you expect. But if you sort by "Last Updated", the sort is actually done correctly, but the "Date" column still reflects the "creation date". I wonder if that's something Rob could tweak? I'm going to open a thread in the "Web Site Issues" section about this.

In reality, it really makes no difference for me during this RDF update job. Call it "paranoia" or (euphemistically?) "attention to detail", but I'm making no real decisions based on dates -- I treat the dates only as a guideline. In reality, I'm planning to carefully compare all variants of a file to see if the changes between 2 or more variants actually make sense. If they don't make sense to me or if I just don't understand, I'm planning to ask (as I've done with Dave, above) and hoping that will get me through the confusion in most cases.

But your point is definitely valid in general and should be taken to heart by anyone updating files in these forums.

Of course, in the long run, I'm really hoping to eliminate the vast majority of files in the "RDF Files" section of the forum, leaving just the "master" zip file of RDFs and anything new since the last release.

Bill

[WagonMaster]

Thanks for the input, Dave. That helps a lot!

Bill based his 15-100 extender RDF on mine, so his works with my map too.

Hmmm... I don't even have an RDF for the extended 15-100. Do you know where that comes from?

Aside: I'd search for the file, but I've found the forum's file search capability to be a bad joke. In fact, I'd recently searched for "15-100" in the "File Search" page, knowing of at least 2 files in the "RDF Files" section that I expected it to find and it came back with nothing. When I put double quotes around the search term, it came back with 43 entries, none of which have "15-100" in the name! I've learned not to rely on the forum's search capabilities.

Does that RDF only come with the extender itself? I'm assuming that I should be putting all extender RDFs in the master file, but if people distribute RDFs with the extenders even after they're in the master RDF file, that's going to add to the confusion.

On the other hand, Rob created the RDF to begin with, and thumbs up/down are used in KM.

I've only run KM once. Even though I don't really intend to use it regularly, I intend to run it more sometime to clear up some of my confusion when things like this come up. But, until then, can you clarify that a bit? Does that mean that Rob would be unable to use the new "master" 15-100 RDF to create a device upgrade with KM? Or is this just a cosmetic issue (i.e. if KM uses button codes and not button names)? If the former, that's a shame because it contributes to the confusion and replication that I'm trying to eliminate. I hope it's the latter.

Bill

[mdavej]

I don't even have an RDF for the extended 15-100. Do you know where that comes from?

All the extenders I know of come with their own custom RDF's. I'd say continue leaving them out of the main distribution since they pretty much have to be created and maintained by the extender author. In the case of the 15-100 extender, Bill and I made sure his RDF continued to work with the same map as the unextended version. That's probably the case with many extender RDF's. The main difference from RM's perspective is the quantity and codes of the phantom keys.

Does that mean that Rob would be unable to use the new "master" 15-100 RDF to create a device upgrade with KM? Or is this just a cosmetic issue (i.e. if KM uses button codes and not button names)? If the former, that's a shame because it contributes to the confusion and replication that I'm trying to eliminate. I hope it's the latter.

AFAIK, KM doesn't use RDF's, Maps or Images at all. So you won't break anything. But I'm sure Mike used an RDF (Rob's in the case of the 15-100) to map the buttons originally in KM. So any differences will be cosmetic.

[vickyg2003]

Now that's interesting. I don't think I noticed that before. I just tried it. If you sort by "Date", it works as expected and the "Date" column is as you expect. But if you sort by "Last Updated", the sort is actually done correctly, but the "Date" column still reflects the "creation date". I wonder if that's something Rob could tweak? I'm going to open a thread in the "Web Site Issues" section about this

Hmm, it seems I was mistaken. I could have sworn I wasn't getting this when I was looking for updates of certain files.

[mr_d_p_gumby]

I don't even have an RDF for the extended 15-100. Do you know where that comes from?

All the extenders I know of come with their own custom RDF's. Most have the same signatures as the unextended versions, some don't. I'd say continue leaving them out of the main distribution since they pretty much have to be created and maintained by the extender author. In the case of the 15-100 extender, Bill and I made sure his RDF continued to work with the same map as the unextended version. That's probably the case with many extender RDF's. The main difference from RM's perspective is the quantity and codes of the phantom keys.

Nils has included extender RDFs in past distributions, so I would not drop the ones that are there already. It would also seem to make sense to add mature extender RDFs so that they can benefit from the review process. IIRC, most extender readme files say to always check the RDF distribution for a later version.

Does that mean that Rob would be unable to use the new "master" 15-100 RDF to create a device upgrade with KM? Or is this just a cosmetic issue (i.e. if KM uses button codes and not button names)? If the former, that's a shame because it contributes to the confusion and replication that I'm trying to eliminate. I hope it's the latter.

AFAIK, KM doesn't use RDF's, Maps or Images at all. So you won't break anything. But I'm sure Mike used an RDF (Rob's in the case of the 15-100) to map the buttons originally in KM. So any differences will be cosmetic.

Correct on all counts. KM does not use the RDF, but I certainly do when I add a remote to KM. The main compatibility issue between KM and the RDF files is the button alias names (not the actual button names). RM can use them to match buttons between a KM file and the RDF. (Each of the official alias names corresponds to a row in a KM file.)