Page 2 of 3

Posted: Sun May 06, 2007 10:36 am
by Darren
Hey Froggi,
you seem to have some good settings thre & I'm too lazy to tinker with mine.
Care to share?

Posted: Sun May 06, 2007 11:03 am
by Rockinfroggi
Darren wrote:Hey Froggi,
you seem to have some good settings thre & I'm too lazy to tinker with mine.
Care to share?
Sure, no problem, I've only gone by things I've read as I don't know enough about all the settings but the changes I have made are:

Prefer all TLDs apart from .COM/.NET/.ORG
Downstream 60%
Upstream 80%
Workers 40
Buckets 30
Archiving Delay 70
LZMA Word size 20

Those are the only changes I've made so far, hope It helps as it would be  good if we could get into the daily top Ten instead of 11th and 12th.


Gary.

Posted: Sun May 06, 2007 11:13 am
by UBT - Halifax-lad
When I get more time to tinker with the settings I may set MJ-12 back to crawling, I was getting too many errors

Posted: Sun May 06, 2007 11:17 am
by Darren
maybe we can post an optimal set of settings so these are my option settings what are your differences....
Image
Image
Image
Image
Image

Posted: Sun May 06, 2007 11:23 am
by Rockinfroggi
UBT - Halifax--lad wrote:When I get more time to tinker with the settings I may set MJ-12 back to crawling, I was getting too many errors

TBH I've pretty much ignored the errors as it will sometimes creep up to high or very high but within an hour or so settles back down to normal, my successes are usually between 70% to 90%.


Gary.

Posted: Sun May 06, 2007 11:34 am
by Rockinfroggi
Darren,

the only difference I can see between yours and mine now is:

Fixed upload chunks: yours 64, mine 32

CPU Overall process: yours normal, mine idle  Main crawl thread, and worker thread: yours Above normal, mine normal.

Archiving process priority: yours high, mine below normal

Posted: Sun May 06, 2007 11:49 am
by Darren
OK, I have adjusted mine to match yours
Lets see if I can improve my speed.

And if Halifax or anyone else would care to help out we could become a top 10 daily team.

Posted: Sun May 06, 2007 11:52 am
by UBT - Halifax-lad
Darren wrote:And if Halifax or anyone else would care to help out we could become a top 10 daily team.
Already back on with same settings, will leave it a day or 2 and see what happens :wink:

Posted: Sun May 06, 2007 2:26 pm
by UBT - Halifax-lad
Now in top 10 for the day :wink:  Bearing in mind I only started back earlier, hopefully when the day starts over again we will be slightly higher :?:

I see Darren has changed his Flag :roll:

Posted: Sun May 06, 2007 6:08 pm
by Rockinfroggi
Yup, just looked, we are #9 at the mo  :D  

Just thought, not sure if my figures are better because I have 3 nodes running? I didn't think that would make any difference as they all have to share the same bandwidth.

Just as a test I'll shut down 2 of my nodes around midnight tonight and run on just one tomorrow to see if my rate drops down.



Gary.

Posted: Sun May 06, 2007 6:10 pm
by UBT - Halifax-lad
you ain't seen nothing yet, think I just remembered my optimum settings, will check for next few days then let you know

Posted: Sun May 06, 2007 6:32 pm
by Rockinfroggi
:shock:  :evil:  :angry4: Iza gonna be watchin you very closely and if ya don't share yer settings Darren will sit on you while we get some brave soul to give you a full body wax very very slowly. :P


Guantanamo Frogster.

Posted: Sun May 06, 2007 6:40 pm
by UBT - Halifax-lad
Rockinfroggi wrote::shock:  :evil:  :angry4: Iza gonna be watchin you very closely and if ya don't share yer settings Darren will sit on you while we get some brave soul to give you a full body wax very very slowly. :P


Guantanamo Frogster.
Put it this way shall I just to annoy you :lol:  I used to be able to crawl 1.5 million URL's in a day when I got the settings perfect, only ever did it for the one day though, I slowed it down after that to less

Posted: Sun May 06, 2007 9:58 pm
by Rockinfroggi
Can I ask (in a French accent) why you have gone with the English flag as opposed to the Uk? If everyone had the UK flag as with all the other projects we would be #4 overall in the world, and if you share your super duper pooper scooper settings we may work our way to #1 in the world.

Just a Frog thinking out load, ignore as required :?


Crunchy Nut Frog (just add milk)

Posted: Sun May 06, 2007 10:06 pm
by UBT - Halifax-lad
Rockinfroggi wrote:Can I ask (in a French accent) why you have gone with the English flag as opposed to the Uk? If everyone had the UK flag as with all the other projects we would be #4 overall in the world, and if you share your super duper pooper scooper settings we may work our way to #1 in the world.

Just a Frog thinking out load, ignore as required :?


Crunchy Nut Frog (just add milk)
Well I am in England :wink:  Not too bothered for the coutry stats as such, as its more our overall stats that will help push us up

Posted: Sun May 06, 2007 10:14 pm
by UBT - The Prof....
Just joined in on this, following setting advice given in everyone's posts. Got to give it a go. Will wait and see what happens..

Posted: Sun May 06, 2007 10:15 pm
by UBT - Halifax-lad
UBT - The Prof.... wrote:Just joined in on this, following setting advice given in everyone's posts. Got to give it a go. Will wait and see what happens..
Cool well any questions just ask

Posted: Sun May 06, 2007 10:17 pm
by UBT - The Prof....
Thanks folks, seems to be running ok so far

Posted: Sun May 06, 2007 10:19 pm
by Rockinfroggi
Ok, changed my flag to Norway as they seem to be the top country  :P


Traitor.

Posted: Sun May 06, 2007 10:20 pm
by UBT - Halifax-lad
Rockinfroggi wrote:Ok, changed my flag to Norway as they seem to be the top country  :P


Traitor.
Gonna come and squash you frog :evil:

Posted: Sun May 06, 2007 10:32 pm
by Rockinfroggi
Dette innebærer at våre publikasjoner formidles til bibliotek, universitet, høyskoler og andre institusjoner der bøkene gjøres tilgjengelige for offentligheten - og selvsagt også til interesserte privatpersoner. Når ikke annet er avtalt avleveres ett eksemplar til hver forespørrer. I tillegg kommer den vanlige pliktavleveringen til Nasjonalbiblioteket. Forlaget har så langt distribuert bøker til Norge  :P

Posted: Sun May 06, 2007 10:38 pm
by UBT - Halifax-lad
Rockinfroggi wrote:Dette innebærer at våre publikasjoner formidles til bibliotek, universitet, høyskoler og andre institusjoner der bøkene gjøres tilgjengelige for offentligheten - og selvsagt også til interesserte privatpersoner. Når ikke annet er avtalt avleveres ett eksemplar til hver forespørrer. I tillegg kommer den vanlige pliktavleveringen til Nasjonalbiblioteket. Forlaget har så langt distribuert bøker til Norge  :P

Code: Select all

This involve that our pamphlets intermediary at directory , college høyskoler and other the institutions there books do accessible for public and obviously also at interested privatpersoner. As nothing but am agreed upon deliver a copy at apiece enquiry. Additional comes it common pliktavleveringen at Nasjonalbiblioteket. Forlaget has so far distributed books at Norway

Posted: Sun May 06, 2007 10:44 pm
by Rockinfroggi
Gawd, your grammar needs working on, no wonder I never have a clue what you're talking about  :lol:



Eric The Green.

Posted: Sun May 06, 2007 10:45 pm
by UBT - Halifax-lad
Rockinfroggi wrote:Gawd, your grammar needs working on, no wonder I never have a clue what you're talking about  :lol:
That on line translators for you :roll:

Posted: Sun May 06, 2007 11:24 pm
by Darren
Now there's 4 of us we are upto 7th for the day!

Posted: Mon May 07, 2007 9:48 am
by UBT - Halifax-lad
Up to 6th place for the day, my settings seem to be working fine, still need more time to play with them before I pass them out to all of you :wink:

I'm beating Iragi frog anyway

Posted: Mon May 07, 2007 9:53 am
by Rockinfroggi
UBT - Halifax--lad wrote:Up to 6th place for the day, my settings seem to be working fine, still need more time to play with them before I pass them out to all of you :wink:

I'm beating Iragi frog anyway

Here in Iraq we do the beating, so get down you western infidel.
All looks pretty level pegging to me but we will see what the rest of the day brings.

Sad Ham Insane.

Posted: Mon May 07, 2007 9:55 am
by UBT - Halifax-lad
Don't forget though your using 3 machines against my 1, and I'm still kicking ure ass :twisted:

Posted: Mon May 07, 2007 9:57 am
by Darren
Anyone brave enough to turn on the webserver bit yet?
I have but have changed it from port 80! (Just in case I do summat stupid with the router)
You get more info that way and also Halifax could see his breakdown of errors.

Posted: Mon May 07, 2007 10:08 am
by Rockinfroggi
UBT - Halifax--lad wrote:Don't forget though your using 3 machines against my 1, and I'm still kicking ure ass :twisted:

Well there's team spirit for you, here I was in my own little world thinking we were out to beat other teams and it turns out we are trying to beat eackother :shock:

Not only that, I now have to protect my donkey from from Laddies boot :?




Anyone brave enough to turn on the webserver bit yet?
Not even looked at that yet but may have a play with it later today.


UnderFrog

Posted: Mon May 07, 2007 10:10 am
by UBT - Halifax-lad
Darren wrote:Anyone brave enough to turn on the webserver bit yet?
I have but have changed it from port 80! (Just in case I do summat stupid with the router)
You get more info that way and also Halifax could see his breakdown of errors.
Don't know how to do that :roll:

Posted: Mon May 07, 2007 10:11 am
by UBT - Halifax-lad
Rockinfroggi wrote:Well there's team spirit for you, here I was in my own little world thinking we were out to beat other teams and it turns out we are trying to beat eackother :shock:
Wait until you get my settings though, you will kick me all over at 3 times the rate :evil:

Posted: Mon May 07, 2007 10:15 am
by Darren
I changed mine to port 85
Ticked the option for webserver
Ticked the option for authenticate
Put in a user and password

restarted
and bobs your uncle

I'll PM the pair of you the settings so you can see mine.
I have done a port forward to allow it!

Posted: Mon May 07, 2007 10:26 am
by UBT - Halifax-lad
If Iraqi Frog would play nice and set his flag to England, then we would also be able to get England into the top 5 countries list for the day :wink:

Posted: Mon May 07, 2007 11:09 am
by UBT - Halifax-lad
Time to show my settings then I think, it seems to be running fine now on my machine, I know I can push it some more as yet

So here is the main display

Image

I am crawling all domains

Image

Here is my connection settings, just realised from looking at screen shot I had not chosen my connection type, so done that now :oops:

Image

My crawl settings, the workers are still quite low and I will push them up further later on during the course of the week. The workers are controlled by profiles. Notice I have the retry boxes ticked on all the DNS & other hard errors, this means before they are classed as an error they will be retried, cuts down the amount of fails.

Image

The next two are profiles, these are great to use and should be used at all times. I can set MJ-12 to work harder when I'm in bed and not using the computer, when I'm at work, etc etc. On the 2nd image the "override number of workers to be" is in actual number of workers not as a percentage as the profile review would have you believe. To enable the override workers function to work correctly you have to ensure that you have set the amount of maximum workers correctly in the crawl options

Eg Max amount of workers in crawl options set to 50 you then cannot ask a profile to have 90 workers as the crawl option only allows 50

I also have an override profile which I can activate if I'm using computer not within my specified hours just in case the crawler makes the net slow, to activate that you have to inactivate the timed profiles you may have set up.  To easily get to profiles once set up to activate or inactivate just right click on MJ-12 icon in sys-tray and go to profiles

Image

Image

Then rest of screens

Image

Image

Posted: Mon May 07, 2007 12:23 pm
by Darren
I have changed mine so it looks more like your settings then halifax.
Are there any TLD's you ain't got checked?

Also what happened to martle?
Is it worth me dropping him an email?

Posted: Mon May 07, 2007 12:31 pm
by UBT - Halifax-lad
Darren wrote:I have changed mine so it looks more like your settings then halifax.
Are there any TLD's you ain't got checked?

Also what happened to martle?
Is it worth me dropping him an email?
I have got it crawling all the domains, thought there is no point missing any out, forgot about Martle, might be an idea to message him and see how he is doing :wink:

Posted: Mon May 07, 2007 3:24 pm
by Darren
Despite getting a bit left behind by you two I have notice that we have now accrued our best days total since the end of October.

Now can we get a top 5 day?
Image

Posted: Mon May 07, 2007 3:27 pm
by UBT - Halifax-lad
Should be quite easy, don't forget you only changed your settings part way through a day, Iraqi Frog has not changed his yet, so tomorrow all been well will be the best, we should get a top 5 for the team each day, for England (if iraqi frog changes over) and hopefully some good individual rankings :wink:

Posted: Mon May 07, 2007 3:33 pm
by Darren
Currently 13th best day! (See above)
My machine will be off from 6.30-16:00 tomorrow.

Not on while I am working!

Posted: Mon May 07, 2007 4:29 pm
by Rockinfroggi
UBT - Halifax--lad wrote:If Iraqi Frog would play nice and set his flag to England, then we would also be able to get England into the top 5 countries list for the day :wink:
Sorry, not been on for a few hours, I shall kill off Sad Ham and fly the flag for England


The patriot.

Posted: Mon May 07, 2007 4:43 pm
by Darren
Then I'll become welsh for now incase Martle pops back in!

Posted: Mon May 07, 2007 5:09 pm
by UBT - Halifax-lad
Darren wrote:Then I'll become welsh for now incase Martle pops back in!
Here is me trying to get England in the top stats, then the numptys arrive and change flags :shock:

Posted: Mon May 07, 2007 5:24 pm
by Rockinfroggi
UBT - Halifax--lad wrote: Well I am in England :wink:  Not too bothered for the coutry stats as such, as its more our overall stats that will help push us up

*Cough*  
Here is me trying to get England in the top stats, then the numptys arrive and change flags Shocked

Make ya bloomin mind up, you're as fickle as a woman :oops:

All settings changed on 2 nodes, thought it might be an idea to leave one node with my original settings to see what improvement there is, if any.


The Quintessential English Frog.

Posted: Mon May 07, 2007 5:26 pm
by UBT - Halifax-lad
Guilty as charged by Lawyer Frog

Posted: Mon May 07, 2007 8:03 pm
by Darren
Must have a chance at our 2nd best total ever...
Image

Posted: Mon May 07, 2007 8:08 pm
by UBT - Halifax-lad
4 hrs until the stats start again for the next day

Posted: Tue May 08, 2007 8:50 am
by Rockinfroggi
Over night, one of my nodes I changed to your settings has 8 buckets and all are returning pretty much 100% time out errors, would that just mean they are all dead sites or is my node throwing a wobbly?

Edit, I've now turned off the retry feature on that node and it's settled down now.

Gary.

Posted: Tue May 08, 2007 5:04 pm
by Darren
Oy,

are all your 3 nodes on the same connection because you are chewing your way through bandwidth like mad today!

Posted: Tue May 08, 2007 7:30 pm
by Rockinfroggi
Piggies are supposed to go Oink not Oy :?

Yes, all nodes are on the same connection, that's why I didn't think it would make that much difference plus looking at yours and David's settings you both have faster connections than me.

I think I would have done better today but one node was returning nothing for half the day until I delete the profiles.

Still, even with all our increased production we seem to have dropped a couple of places  :shock:


Gary.

Posted: Tue May 08, 2007 8:41 pm
by UBT - The Prof....
Advice please.

Think I have this running ok.

Showing 10 buckets, 50+ workers, but also showing errors v.high  :?
Is this ok, or do I need to tweak it a little.
Answers on post cards please to the usual address..  :P

Posted: Wed May 09, 2007 6:14 pm
by Rockinfroggi
Has everyone apart from The prof given up on this?



Gary.

Posted: Wed May 09, 2007 6:51 pm
by UBT - Halifax-lad
Rockinfroggi wrote:Has everyone apart from The prof given up on this?

Gary.
Got bored of it

Posted: Wed May 09, 2007 6:56 pm
by Darren
I just got a warning from my ISP about download levels over the past couple weeks. No doubt this had something to do with it over the last few days.
So I am laying low for a couple weeks but may start it up again once the dust settles.

I can't really afford to be capped again!

Posted: Wed May 09, 2007 7:00 pm
by Rockinfroggi
UBT - Halifax--lad wrote:
Rockinfroggi wrote:Has everyone apart from The prof given up on this?

Gary.
Got bored of it

:roll:  :roll:  :roll:  :roll:  :roll:

Posted: Wed May 09, 2007 7:02 pm
by Rockinfroggi
Darren wrote:I just got a warning from my ISP about download levels over the past couple weeks. No doubt this had something to do with it over the last few days.
So I am laying low for a couple weeks but may start it up again once the dust settles.

I can't really afford to be capped again!

I was just wondering whether capping might be a problem not that I've had a warning yet but I may ease off a bit.


Gary.

Posted: Wed May 09, 2007 7:18 pm
by Darren
Last time I got capped my connection speed between 18:00-23.30 went from 200kB/sec to 5.5kB/sec

It was like the old 56k modem days again.
Took me a few months to get uncapped as well.
All complaint fell on deaf ears.

Posted: Fri May 11, 2007 4:43 pm
by Rockinfroggi
Looks like I'm flying the flag on my own on this now.
Obviously there is no real benefit to the team stats so is there any point in me continuing or is it just a waste of time?


Gary.

Posted: Fri May 11, 2007 6:27 pm
by UBT - The Prof....
Rockinfroggi wrote:Looks like I'm flying the flag on my own on this now.
Obviously there is no real benefit to the team stats so is there any point in me continuing or is it just a waste of time?


Gary.
You are not alone.  I'm still going, though not as fast as some others  :shock:

I think it worthwhile, hope the others come back soon.

Posted: Sat May 12, 2007 12:21 am
by Rockinfroggi
UBT - The Prof.... wrote: You are not alone.  I'm still going, though not as fast as some others  :shock:

I think it worthwhile, hope the others come back soon.

Good man, I'll stick with it for now then as it doesn't affect anything else.


Gary.

Posted: Sat May 12, 2007 4:42 pm
by UBT - Halifax-lad
I'm back again, running it slower than last time as it always slows my comp when I go full blast

Posted: Sat May 12, 2007 6:43 pm
by Rockinfroggi
UBT - Halifax--lad wrote:I'm back again, running it slower than last time as it always slows my comp when I go full blast

Glad to see Prof and I have company, I've throttled back a bit too, just have 2 nodes running for now.


Gary.

Posted: Sat May 12, 2007 9:30 pm
by UBT - The Prof....
I have also slowed this down as its making everything else run slowly, but will stay with it for the time being.  :wink:

Posted: Mon May 14, 2007 5:41 pm
by UBT - Halifax-lad
Due to overtake 3 teams in the next 10 days, thats the easy bit they are not producing anything at all :lol:

Posted: Mon May 14, 2007 5:55 pm
by Rockinfroggi
:roll:

Posted: Wed May 16, 2007 11:55 pm
by UBT - Halifax-lad
Oi, frog how many machines you running on :?

Posted: Thu May 17, 2007 12:31 am
by Active Tech
well i joined up on this one has well hope i'm doing it right tho

Posted: Thu May 17, 2007 12:32 am
by UBT - Halifax-lad
Active Tech wrote:well i joined up on this one has well hope i'm doing it right tho
Welcome to the team, forum & MJ-12, you can't really do it too wrong :lol:  if you go back 2 - 3 pages and then start reading through the messages you will see hints & tips for settings.

Feel free to ask any questions. Do you crunch BOINC then :?:

Posted: Thu May 17, 2007 12:36 am
by Active Tech
yep i'm on Einstein,rosetta,SETI,QMC,Chess960,and ihc

Posted: Thu May 17, 2007 12:59 am
by Rockinfroggi
UBT - Halifax--lad wrote:Oi, frog how many machines you running on :?

Had four of my PC's on this but cut back to 2, why? what's ya problem huh you've been ahead of me most of yesterday :P


Majestic Frog

Posted: Thu May 17, 2007 7:25 am
by UBT - Halifax-lad
Just wondering how my one PC is nearly kicking ure ass all over

Posted: Thu May 17, 2007 7:46 am
by Rockinfroggi
UBT - Halifax--lad wrote:Just wondering how my one PC is nearly kicking ure ass all over

Coz you iz keepin your secret settings secret I should fink  :P



Pond, James Pond 00sieben

Posted: Thu May 17, 2007 7:49 am
by Rockinfroggi
Active Tech wrote:well i joined up on this one has well hope i'm doing it right tho

Welcome Active, good to have you on-board.



Gary.

Posted: Thu May 17, 2007 11:56 am
by Active Tech
thank you  :D

Posted: Thu May 17, 2007 6:08 pm
by Rockinfroggi
UBT - Halifax--lad wrote:Pushing my connection want to see how long it takes a cap to be introduced, should have happened 10 mins ago but still at full speed for now, got a program that monitors how much bandwidth I have used

You wanna be capped? and there was me thinking I was the forum idiot :?


Gary.

Posted: Thu May 17, 2007 6:23 pm
by UBT - Halifax-lad
I can only be capped between 4pm & midnight and then all I get is a 4 hr speed restriction, so no download limit cap as such, just a speed restriction. On my package it would be capped to 2Mbps, so far I have gone 4 times over my limit but still at full speed :wink:

http://allyours.virginmedia.com/html/in ... affic.html

Posted: Thu May 17, 2007 6:30 pm
by Rockinfroggi
Sounds like a pretty fair set up, maybe I should start looking elsewhere as my speed has not been increased in years and although they are talking about doubling it in June it's still going to be way behind othe ISP's.



Gary.

Posted: Thu May 17, 2007 7:10 pm
by UBT - Halifax-lad
6 times over the limit and still no cap :roll:

Posted: Thu May 17, 2007 7:20 pm
by UBT-mark3346
"Sometimes we moderate the speeds", perhaps their reduced customer base has given them some more headroom..
Looks like if they do cap you are capped for 4 hours not just to to the end of peak time "This will last for 4 hours from when the traffic management policy is applied"
Still seems a quite reasonable and open "fair usage" policy :lol:

Posted: Thu May 17, 2007 7:22 pm
by UBT - Halifax-lad
It is quite good really, its not like it stops you using the internet at all. Would be worse if it was a cap on the amount downloaded instead

Posted: Mon May 21, 2007 10:35 am
by Rockinfroggi
I have no idea how but it's 10:30am and I've passed the 1,000,000 mark already today :shock:  I usually struggle to get to that figure in a full day.

Also it has put me at #4 in today's top users.



Gary.

Posted: Sat May 26, 2007 10:42 am
by Rockinfroggi
Think I'm going to change back to the UK flag on this as Mr Halifax has done so and stopped contributing. Also the UK are at #4 at the moment so I think it makes more sense.




Gary.

Posted: Thu Oct 04, 2007 9:31 pm
by melter65
Question for Darren......How often do you update the MJ12 stats? I'm on about 2,000,000 URL's, but my stats are still showing zero! :shock:

Posted: Fri Sep 24, 2010 11:17 pm
by Active Tech
well i'm back on this again but going to do one node for now

lets see how well i will do

Posted: Sat Sep 25, 2010 12:37 pm
by Active Tech
i'm about to hit 1 mil URL's done ...just waitting on this 10k node to be done \o/

AT

Posted: Mon Sep 27, 2010 11:45 am
by Siborg
I've just started running this project at home but I have a pressing question about it. Where is the search engine were helping to build?

Posted: Mon Sep 27, 2010 2:10 pm
by UBT - bobuk
Siborg wrote:I've just started running this project at home but I have a pressing question about it. Where is the search engine were helping to build?
question from me is:

whats the credits like?

winters on the way extra heat will help keep me warm  :wink:

b. :D

Posted: Thu Sep 30, 2010 12:38 am
by melter65
No credits Bob, that's why it's in the 'Other Distributed Computing Projects' section of the forum  :wink:

Posted: Thu Sep 30, 2010 1:37 am
by Siborg
I don't know if I'll continue with this project or not. The actual search engine is never online to use.