Majestic-12

[UBT - Halifax-lad]

I see Rockinfroggi has joined up, well done. Anyone else old or new wish to give it a go

Here is a list of members so far http://www.majestic12.co.uk/projects/ds ... .php?id=67

[Darren]

I started again yesterday but I can not get any speed out of it.
We had a great setup before but I just can seem to get the setting right.

I have told it to use 15 buckets but only 4 start.
Anyone still got our old settings?

[UBT - Halifax-lad]

I started again yesterday but I can not get any speed out of it.
We had a great setup before but I just can seem to get the setting right.

I have told it to use 15 buckets but only 4 start.
Anyone still got our old settings?

Mine is set at 28 works, with 25 buckets, it won't always use the full amount of workers.

Try those settings and see what happens, I'm getting quite a bit done on these settings, its more trial & error with this one. Just keep an the out on the errors text bottom right on main page of the window, if that alters from normal to high or v.high then you need to alter the settings back down, I'm slowly pushing mine up to find my ideal settings.

Don't forget to restart the client if you alter the numbers of workers

[Rockinfroggi]

The client has shut down a few times due to an error but doesn't say what the error is, will slowly read up in the forum and work out what the hell I'm doing

Does it run OK with DepSpid as both use bandwidth?.


Gary.

[UBT - Halifax-lad]

The client has shut down a few times due to an error but doesn't say what the error is, will slowly read up in the forum and work out what the hell I'm doing

Does it run OK with DepSpid as both use bandwidth?.


Gary.

I never had a problem running it before with that or DIMES way back then.

PS: You can alter your flag to English if you wish rather than been a UK flag

[UBT - Halifax-lad]

This may be of use to anyone who is going to use MJ-12

how to crawl faster

[Rockinfroggi]

I never had a problem running it before with that or DIMES way back then.

PS: You can alter your flag to English if you wish rather than been a UK flag

It could be something in my setup or settings so I'll have a read up and a fiddle until I break something.

I used the UK flag because I thought we were a UK team? and as they don't have the flag for my planet 'Alpha Stupidita' :dontknow: would you rather I had the English flag (does it affect the stats?)


Gary.

[UBT - Halifax-lad]

It could be something in my setup or settings so I'll have a read up and a fiddle until I break something.

I used the UK flag because I thought we were a UK team? and as they don't have the flag for my planet 'Alpha Stupidita' :dontknow: would you rather I had the English flag (does it affect the stats?)

If you can't find the problem just put a help request in the forum over there, usually get a prompt reply

No flags don't affect the stats at all, its down to personal preference I suppose, notice we have a welsh dragon amongst us  :lol:

[UBT - Halifax-lad]

I also just spotted this

If you run Windows XP SP2 (Service Pack 2 -- other operating systems are not affected) then you might want to check Events Viewer -> Application. You may notice some of the following warnings:

"TCP/IP has reached the security limit imposed on the number of concurrent TCP connect attempts."

This is so-called Microsoft security kicking in if they detect more than X failed outgoing connections, which can happen with crawler because it can come across with sites that are down.

This "security" slows down crawling and possibly responsible for more timeouts than necessary, so I am searching (still using Google, but hope not for long) for ways to workaround this POS of a feature

More info can be found here

I checked it out how it tells you to check and I was getting lots of those errors so I used this to patch over Microsoft's settings it was very easy, just open the .exe file thats downloaded, it opens up a DOS window, just follow the screen settings, mine is now set to 50. As soon as this is changed windows brings up a warning saying it wants the CD Windows was on, just cancel this as it is Windows trying to alter the settings back

[Rockinfroggi]

Just checked my event logs and under security there is not one error but under Applications I have a bunch of Net Runtime 2.0 Error Reportings and they all tie in with when the client shuts down.


Gary.

[UBT - Halifax-lad]

Just checked my event logs and under security there is not one error but under Applications I have a bunch of Net Runtime 2.0 Error Reportings and they all tie in with when the client shuts down.


Gary.

Stick in a request for help over there thenm with any luck it can be solved, I recall mine used to shut down every so often but it was quite rare for me. I don't think you would see the other warnings unless you increase the settings for workers, I have so I think thats why mine was showing the warnings

[Rockinfroggi]

Touch wood it hasn't shut down for the best part of the day but I increased the number of buckets and workers as I only had 6 buckets running at any one time and instead of getting more it's gone down to 3 buckets?
Yes I did restart the client after changing. No biggie, just thought I'd mention it while I see what else I can break.

Edit, Just a thought, I take it there is nothing to be gained by adding this to all my PC's as they would be sharing the same bandwidth, but would I be better off putting it on one of my dedicated Boinc systems rather than my main workhorse PC?


Bam Bam.

[UBT - Halifax-lad]

I suppose it would be OK with multiple computers on the same connection, give it a try I don't have that privilege here to test it on

[Rockinfroggi]

Just in case anyone else has the problem I had, here's the reply I got from their forum.

Hi there!

I think I know what it is, doh - I forgot to add config file to .NET 2.0 build, you can workaround this as follows:

create text file in node's directory called MJ12node.exe.config with the following text in it:

Code:


<configuration>
<runtime>
<legacyUnhandledExceptionPolicy enabled="1" />
<legacyNullReferenceExceptionPolicy enabled = "1"/>
</runtime>
<system.net>
<settings>
<httpWebRequest useUnsafeHeaderParsing="true" />
</settings>
</system.net>
</configuration>


I am going to update .NET 2.0 ZIP file now: you can alternatively unzip and overwrite all files from here: http://majestic12.co.uk/files/mj12node/ ... 2_v148.zip

Gary.

[UBT - Halifax-lad]

The frog fixes a problem

[Rockinfroggi]

The frog fixes a problem

Me neva dun fixed nuffin, me just found the problem, and anyway, you should celebrate when I do summit right fer a change  


Frog'll Fix It.

[UBT - Halifax-lad]

Frog'll Fix It.

and then manage to break it again even worse that the last time

[Rockinfroggi]

I started again yesterday but I can not get any speed out of it.
We had a great setup before but I just can seem to get the setting right.

I have told it to use 15 buckets but only 4 start.
Anyone still got our old settings?

Mine is set at 28 works, with 25 buckets, it won't always use the full amount of workers.

Try those settings and see what happens, I'm getting quite a bit done on these settings, its more trial & error with this one. Just keep an the out on the errors text bottom right on main page of the window, if that alters from normal to high or v.high then you need to alter the settings back down, I'm slowly pushing mine up to find my ideal settings.

Don't forget to restart the client if you alter the numbers of workers

Like Darren I'm only showing 4 Mrs Bouquets at a time, how many are you showing in Activity David?


Long Frog Kick The Bucket.

[UBT - Halifax-lad]

got 7 open at moment, it won't always open all the buckets you have asked for its normal from what I can gather

[UBT - Halifax-lad]

Having problems now, mainly due to my router from looks of it, causing way too many time outs, so back off MJ-12 again oh well

[Darren]

Hey Froggi,
you seem to have some good settings thre & I'm too lazy to tinker with mine.
Care to share?

[Rockinfroggi]

Hey Froggi,
you seem to have some good settings thre & I'm too lazy to tinker with mine.
Care to share?

Sure, no problem, I've only gone by things I've read as I don't know enough about all the settings but the changes I have made are:

Prefer all TLDs apart from .COM/.NET/.ORG
Downstream 60%
Upstream 80%
Workers 40
Buckets 30
Archiving Delay 70
LZMA Word size 20

Those are the only changes I've made so far, hope It helps as it would be  good if we could get into the daily top Ten instead of 11th and 12th.


Gary.

[UBT - Halifax-lad]

When I get more time to tinker with the settings I may set MJ-12 back to crawling, I was getting too many errors

[Darren]

maybe we can post an optimal set of settings so these are my option settings what are your differences....

[Rockinfroggi]

When I get more time to tinker with the settings I may set MJ-12 back to crawling, I was getting too many errors

TBH I've pretty much ignored the errors as it will sometimes creep up to high or very high but within an hour or so settles back down to normal, my successes are usually between 70% to 90%.


Gary.

[Rockinfroggi]

Darren,

the only difference I can see between yours and mine now is:

Fixed upload chunks: yours 64, mine 32

CPU Overall process: yours normal, mine idle  Main crawl thread, and worker thread: yours Above normal, mine normal.

Archiving process priority: yours high, mine below normal

[Darren]

OK, I have adjusted mine to match yours
Lets see if I can improve my speed.

And if Halifax or anyone else would care to help out we could become a top 10 daily team.

[UBT - Halifax-lad]

And if Halifax or anyone else would care to help out we could become a top 10 daily team.

Already back on with same settings, will leave it a day or 2 and see what happens

[UBT - Halifax-lad]

Now in top 10 for the day  Bearing in mind I only started back earlier, hopefully when the day starts over again we will be slightly higher

I see Darren has changed his Flag

[Rockinfroggi]

Yup, just looked, we are #9 at the mo  :D  

Just thought, not sure if my figures are better because I have 3 nodes running? I didn't think that would make any difference as they all have to share the same bandwidth.

Just as a test I'll shut down 2 of my nodes around midnight tonight and run on just one tomorrow to see if my rate drops down.



Gary.

[UBT - Halifax-lad]

you ain't seen nothing yet, think I just remembered my optimum settings, will check for next few days then let you know

[Rockinfroggi]

 :evil:  :angry4: Iza gonna be watchin you very closely and if ya don't share yer settings Darren will sit on you while we get some brave soul to give you a full body wax very very slowly.


Guantanamo Frogster.

[UBT - Halifax-lad]

 :evil:  :angry4: Iza gonna be watchin you very closely and if ya don't share yer settings Darren will sit on you while we get some brave soul to give you a full body wax very very slowly.


Guantanamo Frogster.

Put it this way shall I just to annoy you  I used to be able to crawl 1.5 million URL's in a day when I got the settings perfect, only ever did it for the one day though, I slowed it down after that to less

[Rockinfroggi]

Can I ask (in a French accent) why you have gone with the English flag as opposed to the Uk? If everyone had the UK flag as with all the other projects we would be #4 overall in the world, and if you share your super duper pooper scooper settings we may work our way to #1 in the world.

Just a Frog thinking out load, ignore as required


Crunchy Nut Frog (just add milk)

[UBT - Halifax-lad]

Can I ask (in a French accent) why you have gone with the English flag as opposed to the Uk? If everyone had the UK flag as with all the other projects we would be #4 overall in the world, and if you share your super duper pooper scooper settings we may work our way to #1 in the world.

Just a Frog thinking out load, ignore as required


Crunchy Nut Frog (just add milk)

Well I am in England  Not too bothered for the coutry stats as such, as its more our overall stats that will help push us up

[UBT - The Prof....]

Just joined in on this, following setting advice given in everyone's posts. Got to give it a go. Will wait and see what happens..

[UBT - Halifax-lad]

Just joined in on this, following setting advice given in everyone's posts. Got to give it a go. Will wait and see what happens..

Cool well any questions just ask

[UBT - The Prof....]

Thanks folks, seems to be running ok so far

[Rockinfroggi]

Ok, changed my flag to Norway as they seem to be the top country  :P


Traitor.

[UBT - Halifax-lad]

Ok, changed my flag to Norway as they seem to be the top country  :P


Traitor.

Gonna come and squash you frog

[Rockinfroggi]

Dette innebærer at våre publikasjoner formidles til bibliotek, universitet, høyskoler og andre institusjoner der bøkene gjøres tilgjengelige for offentligheten - og selvsagt også til interesserte privatpersoner. Når ikke annet er avtalt avleveres ett eksemplar til hver forespørrer. I tillegg kommer den vanlige pliktavleveringen til Nasjonalbiblioteket. Forlaget har så langt distribuert bøker til Norge  :P

[UBT - Halifax-lad]

Dette innebærer at våre publikasjoner formidles til bibliotek, universitet, høyskoler og andre institusjoner der bøkene gjøres tilgjengelige for offentligheten - og selvsagt også til interesserte privatpersoner. Når ikke annet er avtalt avleveres ett eksemplar til hver forespørrer. I tillegg kommer den vanlige pliktavleveringen til Nasjonalbiblioteket. Forlaget har så langt distribuert bøker til Norge  :P

Code: Select all

This involve that our pamphlets intermediary at directory , college høyskoler and other the institutions there books do accessible for public and obviously also at interested privatpersoner. As nothing but am agreed upon deliver a copy at apiece enquiry. Additional comes it common pliktavleveringen at Nasjonalbiblioteket. Forlaget has so far distributed books at Norway

[Rockinfroggi]

Gawd, your grammar needs working on, no wonder I never have a clue what you're talking about  :lol:



Eric The Green.

[UBT - Halifax-lad]

Gawd, your grammar needs working on, no wonder I never have a clue what you're talking about  :lol:

That on line translators for you

[Darren]

Now there's 4 of us we are upto 7th for the day!

[UBT - Halifax-lad]

Up to 6th place for the day, my settings seem to be working fine, still need more time to play with them before I pass them out to all of you

I'm beating Iragi frog anyway

[Rockinfroggi]

Up to 6th place for the day, my settings seem to be working fine, still need more time to play with them before I pass them out to all of you

I'm beating Iragi frog anyway

Here in Iraq we do the beating, so get down you western infidel.
All looks pretty level pegging to me but we will see what the rest of the day brings.

Sad Ham Insane.

[UBT - Halifax-lad]

Don't forget though your using 3 machines against my 1, and I'm still kicking ure ass

[Darren]

Anyone brave enough to turn on the webserver bit yet?
I have but have changed it from port 80! (Just in case I do summat stupid with the router)
You get more info that way and also Halifax could see his breakdown of errors.

[Rockinfroggi]

Don't forget though your using 3 machines against my 1, and I'm still kicking ure ass

Well there's team spirit for you, here I was in my own little world thinking we were out to beat other teams and it turns out we are trying to beat eackother

Not only that, I now have to protect my donkey from from Laddies boot

Anyone brave enough to turn on the webserver bit yet?

Not even looked at that yet but may have a play with it later today.


UnderFrog

[UBT - Halifax-lad]

Anyone brave enough to turn on the webserver bit yet?
I have but have changed it from port 80! (Just in case I do summat stupid with the router)
You get more info that way and also Halifax could see his breakdown of errors.

Don't know how to do that

[UBT - Halifax-lad]

Well there's team spirit for you, here I was in my own little world thinking we were out to beat other teams and it turns out we are trying to beat eackother

Wait until you get my settings though, you will kick me all over at 3 times the rate

[Darren]

I changed mine to port 85
Ticked the option for webserver
Ticked the option for authenticate
Put in a user and password

restarted
and bobs your uncle

I'll PM the pair of you the settings so you can see mine.
I have done a port forward to allow it!

[UBT - Halifax-lad]

If Iraqi Frog would play nice and set his flag to England, then we would also be able to get England into the top 5 countries list for the day

[UBT - Halifax-lad]

Time to show my settings then I think, it seems to be running fine now on my machine, I know I can push it some more as yet

So here is the main display



I am crawling all domains



Here is my connection settings, just realised from looking at screen shot I had not chosen my connection type, so done that now



My crawl settings, the workers are still quite low and I will push them up further later on during the course of the week. The workers are controlled by profiles. Notice I have the retry boxes ticked on all the DNS & other hard errors, this means before they are classed as an error they will be retried, cuts down the amount of fails.



The next two are profiles, these are great to use and should be used at all times. I can set MJ-12 to work harder when I'm in bed and not using the computer, when I'm at work, etc etc. On the 2nd image the "override number of workers to be" is in actual number of workers not as a percentage as the profile review would have you believe. To enable the override workers function to work correctly you have to ensure that you have set the amount of maximum workers correctly in the crawl options

Eg Max amount of workers in crawl options set to 50 you then cannot ask a profile to have 90 workers as the crawl option only allows 50

I also have an override profile which I can activate if I'm using computer not within my specified hours just in case the crawler makes the net slow, to activate that you have to inactivate the timed profiles you may have set up.  To easily get to profiles once set up to activate or inactivate just right click on MJ-12 icon in sys-tray and go to profiles





Then rest of screens

[Darren]

I have changed mine so it looks more like your settings then halifax.
Are there any TLD's you ain't got checked?

Also what happened to martle?
Is it worth me dropping him an email?

[UBT - Halifax-lad]

I have changed mine so it looks more like your settings then halifax.
Are there any TLD's you ain't got checked?

Also what happened to martle?
Is it worth me dropping him an email?

I have got it crawling all the domains, thought there is no point missing any out, forgot about Martle, might be an idea to message him and see how he is doing

[Darren]

Despite getting a bit left behind by you two I have notice that we have now accrued our best days total since the end of October.

Now can we get a top 5 day?

[UBT - Halifax-lad]

Should be quite easy, don't forget you only changed your settings part way through a day, Iraqi Frog has not changed his yet, so tomorrow all been well will be the best, we should get a top 5 for the team each day, for England (if iraqi frog changes over) and hopefully some good individual rankings

[Darren]

Currently 13th best day! (See above)
My machine will be off from 6.30-16:00 tomorrow.

Not on while I am working!

[Rockinfroggi]

If Iraqi Frog would play nice and set his flag to England, then we would also be able to get England into the top 5 countries list for the day

Sorry, not been on for a few hours, I shall kill off Sad Ham and fly the flag for England


The patriot.

[Darren]

Then I'll become welsh for now incase Martle pops back in!

[UBT - Halifax-lad]

Then I'll become welsh for now incase Martle pops back in!

Here is me trying to get England in the top stats, then the numptys arrive and change flags

[Rockinfroggi]

Well I am in England  Not too bothered for the coutry stats as such, as its more our overall stats that will help push us up

*Cough*  

Here is me trying to get England in the top stats, then the numptys arrive and change flags Shocked

Make ya bloomin mind up, you're as fickle as a woman

All settings changed on 2 nodes, thought it might be an idea to leave one node with my original settings to see what improvement there is, if any.


The Quintessential English Frog.

[UBT - Halifax-lad]

Guilty as charged by Lawyer Frog

[Darren]

Must have a chance at our 2nd best total ever...

[UBT - Halifax-lad]

4 hrs until the stats start again for the next day

[Rockinfroggi]

Over night, one of my nodes I changed to your settings has 8 buckets and all are returning pretty much 100% time out errors, would that just mean they are all dead sites or is my node throwing a wobbly?

Edit, I've now turned off the retry feature on that node and it's settled down now.

Gary.

[Darren]

Oy,

are all your 3 nodes on the same connection because you are chewing your way through bandwidth like mad today!

[Rockinfroggi]

Piggies are supposed to go Oink not Oy

Yes, all nodes are on the same connection, that's why I didn't think it would make that much difference plus looking at yours and David's settings you both have faster connections than me.

I think I would have done better today but one node was returning nothing for half the day until I delete the profiles.

Still, even with all our increased production we seem to have dropped a couple of places  :shock:


Gary.

[UBT - The Prof....]

Advice please.

Think I have this running ok.

Showing 10 buckets, 50+ workers, but also showing errors v.high  :?
Is this ok, or do I need to tweak it a little.
Answers on post cards please to the usual address..  :P

[Rockinfroggi]

Has everyone apart from The prof given up on this?



Gary.

[UBT - Halifax-lad]

Has everyone apart from The prof given up on this?

Gary.

Got bored of it

[Darren]

I just got a warning from my ISP about download levels over the past couple weeks. No doubt this had something to do with it over the last few days.
So I am laying low for a couple weeks but may start it up again once the dust settles.

I can't really afford to be capped again!

[Rockinfroggi]

Has everyone apart from The prof given up on this?

Gary.

Got bored of it

 :roll:  :roll:  :roll:  :roll:

[Rockinfroggi]

I just got a warning from my ISP about download levels over the past couple weeks. No doubt this had something to do with it over the last few days.
So I am laying low for a couple weeks but may start it up again once the dust settles.

I can't really afford to be capped again!

I was just wondering whether capping might be a problem not that I've had a warning yet but I may ease off a bit.


Gary.

[Darren]

Last time I got capped my connection speed between 18:00-23.30 went from 200kB/sec to 5.5kB/sec

It was like the old 56k modem days again.
Took me a few months to get uncapped as well.
All complaint fell on deaf ears.

[Rockinfroggi]

Looks like I'm flying the flag on my own on this now.
Obviously there is no real benefit to the team stats so is there any point in me continuing or is it just a waste of time?


Gary.

[UBT - The Prof....]

Looks like I'm flying the flag on my own on this now.
Obviously there is no real benefit to the team stats so is there any point in me continuing or is it just a waste of time?


Gary.

You are not alone.  I'm still going, though not as fast as some others  :shock:

I think it worthwhile, hope the others come back soon.

[Rockinfroggi]

You are not alone.  I'm still going, though not as fast as some others  :shock:

I think it worthwhile, hope the others come back soon.

Good man, I'll stick with it for now then as it doesn't affect anything else.


Gary.

[UBT - Halifax-lad]

I'm back again, running it slower than last time as it always slows my comp when I go full blast

[Rockinfroggi]

I'm back again, running it slower than last time as it always slows my comp when I go full blast

Glad to see Prof and I have company, I've throttled back a bit too, just have 2 nodes running for now.


Gary.

[UBT - The Prof....]

I have also slowed this down as its making everything else run slowly, but will stay with it for the time being.  

[UBT - Halifax-lad]

Due to overtake 3 teams in the next 10 days, thats the easy bit they are not producing anything at all

[Rockinfroggi]

[UBT - Halifax-lad]

Oi, frog how many machines you running on

[Active Tech]

well i joined up on this one has well hope i'm doing it right tho

[UBT - Halifax-lad]

well i joined up on this one has well hope i'm doing it right tho

Welcome to the team, forum & MJ-12, you can't really do it too wrong  if you go back 2 - 3 pages and then start reading through the messages you will see hints & tips for settings.

Feel free to ask any questions. Do you crunch BOINC then

[Active Tech]

yep i'm on Einstein,rosetta,SETI,QMC,Chess960,and ihc

[Rockinfroggi]

Oi, frog how many machines you running on

Had four of my PC's on this but cut back to 2, why? what's ya problem huh you've been ahead of me most of yesterday


Majestic Frog

[UBT - Halifax-lad]

Just wondering how my one PC is nearly kicking ure ass all over

[Rockinfroggi]

Just wondering how my one PC is nearly kicking ure ass all over

Coz you iz keepin your secret settings secret I should fink  :P



Pond, James Pond 00sieben

[Rockinfroggi]

well i joined up on this one has well hope i'm doing it right tho

Welcome Active, good to have you on-board.



Gary.

[Active Tech]

thank you  :D

[Rockinfroggi]

Pushing my connection want to see how long it takes a cap to be introduced, should have happened 10 mins ago but still at full speed for now, got a program that monitors how much bandwidth I have used

You wanna be capped? and there was me thinking I was the forum idiot


Gary.

[UBT - Halifax-lad]

I can only be capped between 4pm & midnight and then all I get is a 4 hr speed restriction, so no download limit cap as such, just a speed restriction. On my package it would be capped to 2Mbps, so far I have gone 4 times over my limit but still at full speed

http://allyours.virginmedia.com/html/in ... affic.html

[Rockinfroggi]

Sounds like a pretty fair set up, maybe I should start looking elsewhere as my speed has not been increased in years and although they are talking about doubling it in June it's still going to be way behind othe ISP's.



Gary.

[UBT - Halifax-lad]

6 times over the limit and still no cap

[UBT-mark3346]

"Sometimes we moderate the speeds", perhaps their reduced customer base has given them some more headroom..
Looks like if they do cap you are capped for 4 hours not just to to the end of peak time "This will last for 4 hours from when the traffic management policy is applied"
Still seems a quite reasonable and open "fair usage" policy

[UBT - Halifax-lad]

It is quite good really, its not like it stops you using the internet at all. Would be worse if it was a cap on the amount downloaded instead