inherit
Korean Supermodel
1
0
1
7,464
Cyonan
2,189
Jul 31, 2016 20:55:30 GMT
July 2016
admin
Cyonan
Mass Effect Trilogy, Dragon Age: Origins, Dragon Age 2, Dragon Age Inquistion, KOTOR, Baldur's Gate, Neverwinter Nights, Jade Empire
Cyonan
|
Post by Cyonan on Aug 1, 2016 4:20:24 GMT
So one of the things I'd like to do in the final days of the BSN Prime is to archive and collect all of the posts made in the ME3 MP forums in the great effort to document the game's inner workings and mechanics.
Most of my posts were on Google docs so I can always re-link them on here, and I should have most of my non Google docs posts.
Thankfully, we have DragonRacer's thread to work with to get links to everything. It's mostly just a copy paste job.
|
|
inherit
3
0
13,409
Pearl
optics cuck
3,898
August 2016
pearl
Top
Mass Effect Trilogy, KOTOR, Mass Effect Andromeda
FatherOfPearl
FatherOfPearl
7,305
3,002
|
Post by Pearl on Aug 1, 2016 4:25:53 GMT
I'll do what I can to help. Any preferred methods (archive.is, google docs, etc)?
|
|
inherit
Korean Supermodel
1
0
1
7,464
Cyonan
2,189
Jul 31, 2016 20:55:30 GMT
July 2016
admin
Cyonan
Mass Effect Trilogy, Dragon Age: Origins, Dragon Age 2, Dragon Age Inquistion, KOTOR, Baldur's Gate, Neverwinter Nights, Jade Empire
Cyonan
|
Post by Cyonan on Aug 1, 2016 4:31:19 GMT
Right now I'll probably mostly use these forums and Google docs.
Later on I've been toying with the idea of creating a website. I've already got 2 personal websites that mostly sit there as testing servers for PHP development for my job right now, so I'll probably re-purpose one of them.
Though that's further down the line closer to when ME:A comes out, and something I'd be undertaking largely by myself that would be more of a blog where I just post random game mechanic stuff.
|
|
Mgamerz
N3
Toolset and Mod Developer for the Mass Effect trilogy
Games: Mass Effect Trilogy
Origin: Mgamerz
Posts: 326 Likes: 1,895
inherit
18
0
1,895
Mgamerz
Toolset and Mod Developer for the Mass Effect trilogy
326
August 2016
mgamerz
Mass Effect Trilogy
Mgamerz
|
Post by Mgamerz on Aug 1, 2016 4:50:03 GMT
I'd suggest building a data miner that can grab the phpbb code. That way you can preserve the formatting. I was thinking of making one once the boards go read only so the format won't change.
|
|
ComradeShepard7
N1
Games: Mass Effect Trilogy, Dragon Age: Origins
Origin: ComradeShepard7
Posts: 13 Likes: 22
inherit
13
0
22
ComradeShepard7
13
August 2016
comradeshepard7
Mass Effect Trilogy, Dragon Age: Origins
ComradeShepard7
|
Post by ComradeShepard7 on Aug 1, 2016 4:53:42 GMT
I've been thinking about trying to run HTTrack on BSN to support a more gradual information extraction. I'm planning on waiting until posting is locked to do this though it may need to be started earlier to avoid losing information to any nuclear exchanges that may happen in BSN's final days.
Haven't actually used this tool before so I don't know how effective it will be.
|
|
Cyberzombie
N3
Reborn
Games: Mass Effect Trilogy, Dragon Age: Origins, KOTOR, Baldur's Gate, Neverwinter Nights, Jade Empire
Origin: GrantMPCredits
Prime Posts: 1
Prime Likes: 1,000,000
Posts: 277 Likes: 576
inherit
19
0
576
Cyberzombie
Reborn
277
August 2016
chesseffect
Mass Effect Trilogy, Dragon Age: Origins, KOTOR, Baldur's Gate, Neverwinter Nights, Jade Empire
GrantMPCredits
1
1,000,000
|
Post by Cyberzombie on Aug 1, 2016 7:05:31 GMT
So I started copy/pasting, but in the wrong sub-forum. Not sure if what I am doing is useful, I thought to just copy the BSN posts linked in the guide into new posts here and then link to them.
|
|
inherit
115
0
2,714
capn233
1,708
August 2016
capn233
Mass Effect Trilogy, Dragon Age: Origins, Dragon Age Inquisition, Mass Effect Andromeda
|
Post by capn233 on Aug 1, 2016 15:22:45 GMT
Is the plan to try and automate this, or will most things need to be manually re-posted?
There are various things in the SP forums that are worth preserving for ME1-ME3 as well.
|
|
Deleted
Deleted Member
Posts: 0
Deleted
inherit
guest@proboards.com
133
0
Deleted
0
January 1970
Deleted
|
Post by Deleted on Aug 1, 2016 16:47:24 GMT
Cyonan ...
Anubis has an app (Razr or Dunvi might have gotten it for him) that allowed him to download DR's thread ... plus some other stuff for DAI fairly easily.
Instead of recreating the wheel, he's willing to send anyone the zip files.
The app itself it probably useful too ... of course, it's possible he got it from you.
Just sayin'
|
|
ComradeShepard7
N1
Games: Mass Effect Trilogy, Dragon Age: Origins
Origin: ComradeShepard7
Posts: 13 Likes: 22
inherit
13
0
22
ComradeShepard7
13
August 2016
comradeshepard7
Mass Effect Trilogy, Dragon Age: Origins
ComradeShepard7
|
Post by ComradeShepard7 on Aug 2, 2016 2:51:12 GMT
I did a test run of HTTrack on the old BSN and it appears to be cooperating. I'll start a full download later this week once some new HDs show up. At the very least this will give us a source for a more gradual migration of data to here, though the smarter people here may have a better way for handling this migration.
|
|
inherit
265
0
Nov 15, 2024 18:18:41 GMT
12,048
Pounce de León
Praise the Justicat!
7,945
August 2016
catastrophy
caustic_agent
|
Post by Pounce de León on Aug 2, 2016 10:19:10 GMT
So the "Cerberus is lazy" thread is gonna be preserved somehow?
|
|
inherit
115
0
2,714
capn233
1,708
August 2016
capn233
Mass Effect Trilogy, Dragon Age: Origins, Dragon Age Inquisition, Mass Effect Andromeda
|
Post by capn233 on Aug 2, 2016 15:08:39 GMT
So the "Cerberus is lazy" thread is gonna be preserved somehow? Polish Phoenix
[broken link]Sadly the images don't seem to be loading  |
|
inherit
265
0
Nov 15, 2024 18:18:41 GMT
12,048
Pounce de León
Praise the Justicat!
7,945
August 2016
catastrophy
caustic_agent
|
Post by Pounce de León on Aug 2, 2016 15:36:55 GMT
Polish Phoenix [broken link]Sadly the images don't seem to be loading Darn. |
|
inherit
192
0
1
randomfoo
2
August 2016
randomfoo
|
Post by randomfoo on Aug 2, 2016 16:04:57 GMT
Was doing some testing, and was pretty easy to parse threads. I'll let others do the full archiving (I'll probably do some of my own, limited pulls for stuff I care about after the lock-down) but if you have the pages, here's how to get clean data out: github.com/lhl/bsn-crawler/blob/master/test.py |
|
inherit
44
0
8,750
MPApr2012
...look my eyes are just holograms
1,290
August 2016
mpapr2012
|
Post by MPApr2012 on Aug 4, 2016 23:04:01 GMT
hm... not much activity here by the looks of it. grabbed the HTTrack and it seems to be doing... something, will give it a shot with a content-rich thread like Cerberus is Lazy next...
Also, the forums are... lagging. Apologise if I spammed into into your notifications feeds, could not figure out while like button did not work, so kept clicking... and clicking... and clicking
|
|
Mgamerz
N3
Toolset and Mod Developer for the Mass Effect trilogy
Games: Mass Effect Trilogy
Origin: Mgamerz
Posts: 326 Likes: 1,895
inherit
18
0
1,895
Mgamerz
Toolset and Mod Developer for the Mass Effect trilogy
326
August 2016
mgamerz
Mass Effect Trilogy
Mgamerz
|
Post by Mgamerz on Aug 5, 2016 15:19:46 GMT
If someone could run this over the ModMaker thread... 😎
|
|
ComradeShepard7
N1
Games: Mass Effect Trilogy, Dragon Age: Origins
Origin: ComradeShepard7
Posts: 13 Likes: 22
inherit
13
0
22
ComradeShepard7
13
August 2016
comradeshepard7
Mass Effect Trilogy, Dragon Age: Origins
ComradeShepard7
|
Post by ComradeShepard7 on Aug 6, 2016 2:07:33 GMT
This is what my progress looks like ~ 24 hours into running HTTrack on BSN Prime:  For those not familiar with HTTrack, the "links scanned" meaning can be found at the following: httrack.kauler.com/help/Links_scannedSeeing as I have no idea how much is left / how long the full run will take, it may be good to have someone else do more targeted backups (like what MPApr2012 mentioned). |
|
ComradeShepard7
N1
Games: Mass Effect Trilogy, Dragon Age: Origins
Origin: ComradeShepard7
Posts: 13 Likes: 22
inherit
13
0
22
ComradeShepard7
13
August 2016
comradeshepard7
Mass Effect Trilogy, Dragon Age: Origins
ComradeShepard7
|
Post by ComradeShepard7 on Aug 11, 2016 4:29:06 GMT
Got this from HTTrack:
Panic: Too many URLs, giving up..(>100000)
It could still be useful for creating backups of specific threads but that will require crawling through the forum to see what should be selected for backup. If any of the bright minds can point me in a direction where I can assist with whatever efforts are going on I'd be happy to lend a hand. Otherwise I'll just keep doing my own thing.
|
|
