Multiple Custom Mirror

[jamincollins]

The recent 1.4.0 client added support for centralized custom look updates from custom mirror servers, listed in custom_mirrors.lst. However, with the way this is implemented in the client it is not really possible to use multiple different custom servers. First if there are multiple servers, the code randomly selects one of the servers to use and stops after successful communication with any one of the custom servers. This would be fine if all the custom servers were mirrors of each other. However, this is unlikely to be the case and there has already been some contention between the existing service provided by LabRat and myself.

 

RallosZek and I have worked on a patch to keep the existing method of update for the official client updates and utilize the custom_mirrors.lst as a sequential list of servers to check for updates from. The patch can be found here.

[LabRat]

I posted a patch a while back about the custom clothing, and was requested by Learner to communicate them directly to him instead of posting them on berlios (or in the same vein on sites the devs have no write access to), as he and I have been the lead custom clothing programmers, and understand each other's methodology and reasons for checks etc

- if(++update_attempt_count < 3){

+ if(++update_attempt_count < 10){ /* No idea what this check is for */

If you don't understand the reason for something, leaving well enough alone until you either ask what the line is for or think about it and figure it out may be a good thing all round.

However, this is unlikely to be the case and there has already been some contention between the existing service provided by LabRat and myself.

I never even knew that you had a custom clothing mirror until I saw your site asgardsrealm.net repeatedly hammering my site (a failed takedown attempt?) and repeatedly requesting the same files and for that reason I banned your domain and local subnets from my site. Not contentious, purely looking after my own interests - after all, how many copies of erstad's cape1_brown.bmp would you possibly need?

 

As for the custom update sites being mirrors of each other, I know of two other ones, and they are in the main mirrors of each other. We may be a little out of sync but I am sure we will get round to resyncing at some point in the near future.

[jamincollins]

I posted a patch a while back about the custom clothing, and was requested by Learner to communicate them directly to him instead of posting them on berlios (or in the same vein on sites the devs have no write access to), as he and I have been the lead custom clothing programmers, and understand each other's methodology and reasons for checks etc

- if(++update_attempt_count < 3){

+ if(++update_attempt_count < 10){ /* No idea what this check is for */

If you don't understand the reason for something, leaving well enough alone until you either ask what the line is for or think about it and figure it out may be a good thing all round.

I was not aware of any such request concerning patches. So, this seemed the appropriate place to let the developers know of a patch.

 

The comment was placed there by RallosZek, I did not remove the comment when I made my change. I found that the update_attempt_count was causing a very small limitation during my testing. Once the custom_mirrors.lst was treated sequentially, it would only process two entries from the custom file because the primary mirror and the first two from custom caused this variable to exceed the threshold.

 

However, this is unlikely to be the case and there has already been some contention between the existing service provided by LabRat and myself.

I never even knew that you had a custom clothing mirror until I saw your site asgardsrealm.net repeatedly hammering my site (a failed takedown attempt?) and repeatedly requesting the same files and for that reason I banned your domain and local subnets from my site. Not contentious, purely looking after my own interests - after all, how many copies of erstad's cape1_brown.bmp would you possibly need?

The repeated requests were from implementing and testing the code for the provided patch. I'm sorry that C/C++ isn't a strong language for me.

 

In tracing through the code, it became necessary for me to see the code functioning in a retrieval an non-retrieval manner.

 

As for the custom update sites being mirrors of each other, I know of two other ones, and they are in the main mirrors of each other. We may be a little out of sync but I am sure we will get round to resyncing at some point in the near future.

Nothing in my proposed change would prevent that two sites from mirroring each other. Though it would make the first one preferred.

[RallosZek]

I posted a patch a while back about the custom clothing, and was requested by Learner to communicate them directly to him instead of posting them on berlios (or in the same vein on sites the devs have no write access to), as he and I have been the lead custom clothing programmers, and understand each other's methodology and reasons for checks etc

 

Sorry, I didn't see a header/readme describing this procedure.

 

- if(++update_attempt_count < 3){

+ if(++update_attempt_count < 10){ /* No idea what this check is for */

If you don't understand the reason for something, leaving well enough alone until you either ask what the line is for or think about it and figure it out may be a good thing all round.

That comment was made by me, in development-stage and didn't delete it.

But to answer your question: It limits the total ammounts of updates done, following the rationale that all servers are mirrors, and thus only 2 consumers of this functionality will ever pass through there. However, this patch provides the ability to add custom clothing, from different providers, which are assumed not to be mirrors, which is a different rationale. The 10 is currently arbitrary, it should really be 1+(num_update_custom_servers), but we tested with 3 servers so it sufficed.

 

As for the custom update sites being mirrors of each other, I know of two other ones, and they are in the main mirrors of each other. We may be a little out of sync but I am sure we will get round to resyncing at some point in the near future.

 

Yes, but the point is: this patch allows people to use their own servers. If however, this is not desirable behavior, then I do not understand why there is no custom_mirrors.lst provided by official el distribution, so that custom clothing actually works outof-the-box. That's why we worked on a patch that allows people to specify the update server, so that guilds/friends can at least see eachother's own clothing.

Edited June 5, 2007 by RallosZek

[Learner]

The custom mirrors list is supposed to be s list of systems that have identical information. The idea is to try a small number of times to find ONE mirror that it can check itself against. It is not supposed to be a list of servers which all have to be hit to get data from!

[jamincollins]

The custom mirrors list is supposed to be s list of systems that have identical information. The idea is to try a small number of times to find ONE mirror that it can check itself against. It is not supposed to be a list of servers which all have to be hit to get data from!

The problem with that approach is that you then give a select few an effective monopoly on the custom images since a player would need to select which one group of servers to get the images from. Then guilds could not host their own image servers or players would be reluctant to have their images hosted by service Y because other images they would like to also get are hosted by service X.

 

In short the existing method will pretty much ensure that the first service on the market has a perpetual monopoly.

[Learner]

The custom mirrors list is supposed to be s list of systems that have identical information. The idea is to try a small number of times to find ONE mirror that it can check itself against. It is not supposed to be a list of servers which all have to be hit to get data from!

The problem with that approach is that you then give a select few an effective monopoly on the custom images since a player would need to select which one group of servers to get the images from. Then guilds could not host their own image servers or players would be reluctant to have their images hosted by service Y because other images they would like to also get are hosted by service X.

 

In short the existing method will pretty much ensure that the first service on the market has a perpetual monopoly.

The intent for it is to allow Entropy & Roja to setup one or more mirrors that could be seperate from game updates if they want. It is NOT intended for anyone to or a bunch of people to just start producing their own stuff! Anyone that produces their own stuff will have to have their server used INSTEAD of the official EL setup that Roja will be using.

[ttlanhil]

so the problem is that you want to use the same system that will be officially used by EL for updates to custom data?

 

ed: darnit, learner beat me

Edited June 6, 2007 by ttlanhil

[jamincollins]

What I'd like to be able to do is what LabRat is already doing. Provide other players with a service. If they would like custom clothing, they would then have more choices.

 

As I understand it, LabRat's service is ok with Radu and Roja. However, if this feature remains as-is, then players will have to choose between Radu/Roja's service, LabRat's service, or some other service.

 

If the feature is changed to use all the entries, the load on identical mirrors would be marginal. The client would download the list, find that it had the listed files, check the MD5SUM and move on to the next file or server listed. Randomizing the read array of servers read from custom_mirrors would help alleviate the problem of the first server in a synchronized set being the one always hit.

 

This feature is already being used as I would like to use it. I'd just like to see it enhanced a bit.

[Learner]

What I'd like to be able to do is what LabRat is already doing. Provide other players with a service. If they would like custom clothing, they would then have more choices.

 

As I understand it, LabRat's service is ok with Radu and Roja. However, if this feature remains as-is, then players will have to choose between Radu/Roja's service, LabRat's service, or some other service.

 

If the feature is changed to use all the entries, the load on identical mirrors would be marginal. The client would download the list, find that it had the listed files, check the MD5SUM and move on to the next file or server listed. Randomizing the read array of servers read from custom_mirrors would help alleviate the problem of the first server in a synchronized set being the one always hit.

 

This feature is already being used as I would like to use it. I'd just like to see it enhanced a bit.

That can't be allowed or improper content will be in the game or servers fighting over the same content. It stays to be a list of servers that are supposed to have the same content. The reason it's not part of the normal auto update is auto update forces a restart, custom update doesn't.

[Entropy]

As Learner explained, we NEED to have some level of control about what goes into the custom clothes. We don't want a penis cape, with a big dick in erection on it. Or some racist stuff, etc.

 

This system was not intended for everyone to be able to have their own mirror.

However, it is designed in such a manner that if a group of players does not care about potentially inapropriate clothes, they can use their own servers and do whatever they want. Doing so would, of course, be their own business, and anyone complaining to the mods would be promptly warned and possibly banned.

[RallosZek]

Hmm, so basically that the system didn't work outof the box with 1.4.0 was an oversight and because autoupdate and customupdate with it, was disabled during RC, nobody picked up on it.

During testing this, I also noticed that directories were never created (neither tmp nor custom and directories below), I'll check this some more why this is.

 

However, it is designed in such a manner that if a group of players does not care about potentially inapropriate clothes, they can use their own servers and do whatever they want.

Well, currently, it is designed that they have choice of either using their own or the official. If we provide a patch that enables both (official and your own, by using for instance O(fficial): servername and U(noffical): servername), will this be considered or is this not preferable either?

[LabRat]

The only sticking point is the tmp/ folder, once that is created everything else works as it should.

 

I just looked into it, and realised immediately why:

updates.c:handle_update_download:0127

			sts= rename("./tmp/temp000.dat", files_lst);

updates.c:handle_update_download:0175

		// failsafe, try to make sure the directory is there
	if(mkdir_res < 0){
		mkdir_res= mkdir_tree("./tmp");
	}

the code from lines 175 through 178 needs to be moved *before* line 127 otherwise there is no possibility of ever creating files.lst

 

Can't believe I couldn't figure out why I had to create tmp manually before now

Edited June 6, 2007 by LabRat

[Learner]

Hmm, so basically that the system didn't work outof the box with 1.4.0 was an oversight and because autoupdate and customupdate with it, was disabled during RC, nobody picked up on it.

During testing this, I also noticed that directories were never created (neither tmp nor custom and directories below), I'll check this some more why this is.

 

However, it is designed in such a manner that if a group of players does not care about potentially inapropriate clothes, they can use their own servers and do whatever they want.

Well, currently, it is designed that they have choice of either using their own or the official. If we provide a patch that enables both (official and your own, by using for instance O(fficial): servername and U(noffical): servername), will this be considered or is this not preferable either?

It will NOT be considered because we do not want it doing things like downloading from multiple official servers or downloading the same filess everytime you login because one or more files is in the official server and someone elses private server.

[Learner]

The only sticking point is the tmp/ folder, once that is created everything else works as it should.

 

I just looked into it, and realised immediately why:

updates.c:handle_update_download:0127

			sts= rename("./tmp/temp000.dat", files_lst);

updates.c:handle_update_download:0175

		// failsafe, try to make sure the directory is there
	if(mkdir_res < 0){
		mkdir_res= mkdir_tree("./tmp");
	}

the code from lines 175 through 178 needs to be moved *before* line 127 otherwise there is no possibility of ever creating files.lst

 

Can't believe I couldn't figure out why I had to create tmp manually before now

The first time handle_update_download() is called is with a NULL parameter, so the first block of code is skipped, and then it hits the second block.

 

That said, with the current code, it would make more sense if tmp dir check was in init_update before that routine was called.

[LabRat]

ttl and I brainstormed a little, the reason the dir wasn't being created was because of

mkdir_res= mkdir_tree("./tmp");

it needs to read

mkdir_res= mkdir_tree("./tmp/");

with the trailing / or the directory never gets created.

 

init_update() would be disabled if auto_update is disabled, the same is true of init_custom_update() and custom_update is disabled, hence choosing the handle_update_download() function as the home.

Edited June 6, 2007 by LabRat

[Learner]

ttl and I brainstormed a little, the reason the dir wasn't being created was because of

mkdir_res= mkdir_tree("./tmp");

it needs to read

mkdir_res= mkdir_tree("./tmp/");

with the trailing / or the directory never gets created.

 

init_update() would be disabled if auto_update is disabled, the same is true of init_custom_update() and custom_update is disabled, hence choosing the handle_update_download() function as the home.

Why mkdir_tree instead of mkdir? Since it's a hard coded path a simple mkdir should work.

 

I'm guessing it used to be mkdir and then was change to mkdir tree. With mkdir you don't need the trailing / . Yes, the original code used #ifdef WINDOWS to know whiche version of mkdir it needed, and under Linux it made it mode 0777.

[LabRat]

Yeah the mkdir_tree was your work, way back when for the reason you just stated.

 

I just did a full clean install with the latest CVS, it does the funky update stuff as it should now \o/

[RallosZek]

Hmm, so basically that the system didn't work outof the box with 1.4.0 was an oversight and because autoupdate and customupdate with it, was disabled during RC, nobody picked up on it.

During testing this, I also noticed that directories were never created (neither tmp nor custom and directories below), I'll check this some more why this is.

 

However, it is designed in such a manner that if a group of players does not care about potentially inapropriate clothes, they can use their own servers and do whatever they want.

Well, currently, it is designed that they have choice of either using their own or the official. If we provide a patch that enables both (official and your own, by using for instance O(fficial): servername and U(noffical): servername), will this be considered or is this not preferable either?

It will NOT be considered because we do not want it doing things like downloading from multiple official servers or downloading the same filess everytime you login because one or more files is in the official server and someone elses private server.

 

Yeah, my idea was:

custom_mirrors.lst:

O: www.eternal-lands.com

O: mirror.eternal-lands.com

U: www.myguild.com

U: www.allies.com

 

In Pseudo:

if(official) {
  use_mirror_selector();
  official_files[numfiles++] = currently_downloaded_file;
}
else
{
  iterate_servers();
  if(!in_array(scheduled_for_download, official_files))
  {
   download(scheduled_for_download);
  }
}

Which is basically everything we want and everything you don't want worked out. But whatever. It's not a feature that has personal interest, so I won't dwell on it.

[jamincollins]

As Learner explained, we NEED to have some level of control about what goes into the custom clothes. We don't want a penis cape, with a big dick in erection on it. Or some racist stuff, etc.

 

This system was not intended for everyone to be able to have their own mirror.

However, it is designed in such a manner that if a group of players does not care about potentially inapropriate clothes, they can use their own servers and do whatever they want. Doing so would, of course, be their own business, and anyone complaining to the mods would be promptly warned and possibly banned.

 

So, if a custom mirror provider (such as myself, LabRat, or others) decided to mirror the official server and supplement it with additional items for ourselves or others, would this be acceptable. I don't expect it to be "official", I'm just looking for a way for people to be able to get the images that the official server would have and also have some additional choice. This would also reduce the load on the official servers because those electing to use the player provided servers would not be hitting the official servers.

[Aislinn]

As Learner explained, we NEED to have some level of control about what goes into the custom clothes. We don't want a penis cape, with a big dick in erection on it. Or some racist stuff, etc.

 

This system was not intended for everyone to be able to have their own mirror.

However, it is designed in such a manner that if a group of players does not care about potentially inapropriate clothes, they can use their own servers and do whatever they want. Doing so would, of course, be their own business, and anyone complaining to the mods would be promptly warned and possibly banned.

What happens when this occurs and they go around posting the images or url's to the images? This would give EL a bad name imo.

[kl4Uz]

 

What happens when this occurs and they go around posting the images or url's to the images? This would give EL a bad name imo.

 

You can already do this right now. Many people use custom texture files.

[Aislinn]

 

What happens when this occurs and they go around posting the images or url's to the images? This would give EL a bad name imo.

 

You can already do this right now. Many people use custom texture files.

Yes I know but it's not officially supported. (Unapproved ones I mean).

[Roja]

As Learner explained, we NEED to have some level of control about what goes into the custom clothes. We don't want a penis cape, with a big dick in erection on it. Or some racist stuff, etc.

 

This system was not intended for everyone to be able to have their own mirror.

However, it is designed in such a manner that if a group of players does not care about potentially inapropriate clothes, they can use their own servers and do whatever they want. Doing so would, of course, be their own business, and anyone complaining to the mods would be promptly warned and possibly banned.

What happens when this occurs and they go around posting the images or url's to the images? This would give EL a bad name imo.

 

If they are putting bad image on items in game they will simply be banned.

[Learner]

Hmm, so basically that the system didn't work outof the box with 1.4.0 was an oversight and because autoupdate and customupdate with it, was disabled during RC, nobody picked up on it.

During testing this, I also noticed that directories were never created (neither tmp nor custom and directories below), I'll check this some more why this is.

 

However, it is designed in such a manner that if a group of players does not care about potentially inapropriate clothes, they can use their own servers and do whatever they want.

Well, currently, it is designed that they have choice of either using their own or the official. If we provide a patch that enables both (official and your own, by using for instance O(fficial): servername and U(noffical): servername), will this be considered or is this not preferable either?

It will NOT be considered because we do not want it doing things like downloading from multiple official servers or downloading the same filess everytime you login because one or more files is in the official server and someone elses private server.

 

Yeah, my idea was:

custom_mirrors.lst:

O: www.eternal-lands.com

O: mirror.eternal-lands.com

U: www.myguild.com

U: www.allies.com

 

In Pseudo:

if(official) {
  use_mirror_selector();
  official_files[numfiles++] = currently_downloaded_file;
}
else
{
  iterate_servers();
  if(!in_array(scheduled_for_download, official_files))
  {
   download(scheduled_for_download);
  }
}

Which is basically everything we want and everything you don't want worked out. But whatever. It's not a feature that has personal interest, so I won't dwell on it.

The will be NO SUPPORT FOR UNOFFICIAL CUSTOM DOWNLOADS allowed to be added to the official client!

 

What you've outlined will cause excess load on the official servers under several situations as well.