Standardized item data format?
Mon, 2011-10-17 16:42
So one thing I noticed in the homebrew forum a lot is different char sheets, item lists, and proposed computer programs/scripts which rely on gear and morph statistics... I was wondering if there would be any utility in creating a project in google which was just a user-updated gear/morph/implant database in a common, agreed upon format that any developers could just scrape into their own apps or sheets?
The utility for this would be a standardized repository online that can be kept up to date and imported into whatever projects are going to need all this distilled data. A big plain-text database for all.
For example, a good solution for any web-based applications might be a JSON markup like this:
// Laser Pulser
{ 'Laser Pulser' : [
{ 'type' : 'Beam Weapon' },
{'metadata' : [
{ 'book' : 'Core' },
{ 'page' : '338' },
{ 'desc' : '' },
{ 'cost' : 'mod' }
]},
{'properties' : [
{ 'ap' : '0' },
{ 'ammo' : '100' },
{ 'modes' : [1,0,0] }, // boolean array [sa,bf,fa]
{ 'ranges' : [30,100,150,250] }, // int array [short,med,long,extr]
]}
]}
which most modern web browsers just magicaly turn into a working javascript object that can be polled and manipulated.
I feel like, if we can agree on a standard schema and get the data up in the cloud, it might make game tool development a little less intimidating. If we took it a step further, we could even store calculations and relationships between objects for reference or straight usage... not really sure if it's practical, but a neat idea...:
// Lucidity Calculation
{ 'Lucidity' : [
{ 'type' : 'Calculation' },
{'metadata' : [
{ 'book' : 'Core' },
{ 'page' : '122' },
{ 'desc' : 'Lucidity determines how much stress (mental damage) you can take before you are incapacitated or driven insane' },
]},
{'properties' : [
{ 'math' : '(%Will%*2) + %EgoTraits% + %MorphTraits% + %Gear%' }, // %var% denotes variable name
]}
]}
While I appreciate that this kind of thing is better suited to an actual database, that severely limits the accesability IMHO...
[edit] .csv format files are uploaded here:
http://itsmax.net/pub/EclipsePhase/data
Mon, 2011-10-17 18:28
#2
Re: Standardized item data format?
I'd definitly be interested in discussing what kind of output would be useful to whom. If there were some way to keep the darkcast wiki up to date with all of this and just have it vomit up its data, that would be awesome.
I see JSON being useful for web and app devs, but spreadsheet driven tools would more likely need xml or even a simpler csv format...
Ultimately, what I would love to see is a sort of ecosystem where raw data is updated in one place, pulled down to update tools which can in turn output standard formats to upload back to the cloud, or import into other tools... I feel like, if we had that capability, we could also start accumulating a library of custom morphs, pre-build (n)pcs, special modified weapons, etc. which would be easy to just download and plug in... like blueprints in a fabber!
But all of this sort of depends on how many disparate projects are trying to use the data. If someone would just make a single massive character creator, initiative tracker, campaign manager, then we wouldn't really need it. 
Mon, 2011-10-17 23:18
#3
Re: Standardized item data format?
is it for online character sheets or gear codex (no, Felicia, not you!)
I would very interested in either, but I'm totally useless where it comes to code.
__________________
"This is how a human dies! At Ramming speed!"
ANN Reporter Emily Wong, RIP 2186
Q U I N C E Y ^_*_^ F O R D E R
"This is how a human dies! At Ramming speed!"
ANN Reporter Emily Wong, RIP 2186
Tue, 2011-10-18 10:47
#4
Re: Standardized item data format?
Might I suggest adding the revision of the book to the schema to take into account changes in stats?
// Laser Pulser
{ 'Laser Pulser' : [
{ 'type' : 'Beam Weapon' },
{'metadata' : [
{ 'book' : 'Core' },
{ 'revision' : '3' },
{ 'page' : '338' },
{ 'desc' : '' },
{ 'cost' : 'mod' }
]},
{'properties' : [
{ 'ap' : '0' },
{ 'ammo' : '100' },
{ 'modes' : [1,0,0] }, // boolean array [sa,bf,fa]
{ 'ranges' : [30,100,150,250] }, // int array [short,med,long,extr]
]}
]}
__________________
I am everywhere.
Tue, 2011-10-18 16:04
#6
Re: Standardized item data format?
In theory, either and both! It's just raw data in a standardized format that everyone can use for whatever they want, but it would not be in a human friendly format until someone parses it into an app or spreadsheet...
I'd love to have an EP app on my ecto...I mean on my iPad!
That would be awesome!
__________________
"This is how a human dies! At Ramming speed!"
ANN Reporter Emily Wong, RIP 2186
Q U I N C E Y ^_*_^ F O R D E R
"This is how a human dies! At Ramming speed!"
ANN Reporter Emily Wong, RIP 2186
Tue, 2011-10-18 17:23
#7
Re: Standardized item data format?
It's worth noting, once the data fields are determined and the data is in place on the system... then the output can be in any format needed. (At least with Drupal it's that kind of flexible). So if we can make a good solid list of fields for each type of data, then we can pull it out in any form requested afterwards.
Tue, 2011-10-18 17:29
#8
Re: Standardized item data format?
While I like the idea of a unique id, it would have to be automated to avoid collision. If it was being pulled out of the darkcast wiki, I think that a unique id would be a field in one of the database tables already.
You are right - Drupal assigns a unique id to everything in it's system.
Ideally, we're talking about fielded data here though, so one thing to consider is that the input for this would be very different than it is for say a normal wiki post or a forum post. It would be a form where each value to go into the system is clearly defined by the type of data going in. So you pick a morph, it gives you the fields for a morph. You pick a skill, it gives you the fields for a skill - etc etc.
I could also set Drupal up to read in from a spreadsheet I upload - so if you want to collect the initial batch of data on a google doc then that's viable. Basically - we can totally do this 
But we gotta have people who want to do the data entry.
Tue, 2011-10-18 19:15
#9
Re: Standardized item data format?
... It would be a form where each value to go into the system is clearly defined by the type of data going in. So you pick a morph, it gives you the fields for a morph. You pick a skill, it gives you the fields for a skill - etc etc...
I definitely like the form-fill idea, but what about editing currently existing data in the event of typos or updates? Could that be done through the normal wiki interface?
I don't know Drupal, but if I can help make forms templates, I will. We just have to figure out what categories and fields we need.
I could also set Drupal up to read in from a spreadsheet I upload - so if you want to collect the initial batch of data on a google doc then that's viable. Basically - we can totally do this But we gotta have people who want to do the data entry.
But we gotta have people who want to do the data entry. I think there is already a gear listing in spreadsheet form somewhere floating around... at the very least, I know the jmobius character generator excell sheet has all that data hidden in there *somewhere*. If we could get Kindalas to help us find it, we might be able to scrape it in with a little tweaking.
If not, we can probably crowd-source a lot of it. I'd pitch in... say doing all the traits or weapons. This kind of wheel-reinventing is what I was hoping we can avoid in the future.
... if we can make a good solid list of fields for each type of data, then we can pull it out in any form requested afterwards.
So then the next step, if we are going to try to move forward on it, would be to establish how we are organizing our information INTO the darkcast wiki so that we can make it easy to get back out again.
Tue, 2011-10-18 19:45
#10
Re: Standardized item data format?
Thankfully drupal can provide an edit form for any type of data it has. So fixing typos after the fact, or adding new stuff as books come out will be comparatively easy so long as we keep it up to day - it's this initial data load that's hard.
edit: I also just checked the options available for the freelinking module that I use on darkcast - it's what allows for wiki style links using [[ and ]]. It'll work with multiple node types, so we can have the wiki and the db type nodes work together pretty much seamlessly.
Start with:
a) A list of types of data to store
b) A list of the fields within that data to
And I'll construct the content types, forms, and give you the interface from Darkcast. If you want to use a spreadsheet for the initial load - if that's easier to work with - then just make sure each field gets it's own column. I can load it into the wiki when you've got a batch ready to go.
Thu, 2011-10-20 17:43
#11
Re: Standardized item data format?
Ok, I found an HTML version of Core book that was structured enough I could scrape it for data. I currently have a *lot* of the information from Core uploaded:
http://www.firewall-darkcast.com/wiki/category/core
Still to go - Sleights, Backgrounds and Factions. And probably tons of interface issues to make sure the wiki interface works together. Once this thing is in place, I'll start adding ways to pull the information down in various formats.
Thu, 2011-10-20 18:18
#12
Re: Standardized item data format?
For the Excel Character generator I have the following variables for the morphs
Name: this would be a string
Aptitude Max: number
CP Cost: number
Credit Cost: number
COG: bonus to COG
COO: bonus to COO
INT: bonus to INT
REF: bonus to REF
SAV: bonus to SAV
SOM: bonus to SOM
WIL: bonus to WIL
Pick: from 0 to 3 enables a dropdown to choosing which aptitude gets a +5 without dupilcate bonuses
-5 to a skill: 0 or 1 gives a dropdown
Durability: just a number
WT: actually a useless variable that is autocalculated
Synthetic?: 0 or 1 with 1 meaning synthetic
pod?: 0 or 1 with 1 meaning pod constuction
Uplift?: 0 or 1 with 1 meaning an uplift
S1: bonus skill one's name
S#: bonus skill one's value
S2: bonus skill two's name
S#: bonus skill two's value
Gear: this is a list of "gearname, gearname, gearname"
Traits: this is a list of "morphtrait, morphtrait (level #), morph trait"
Energy Armor: just a number
Kinetic Armor: just a number
description: just text
Wed, 2011-10-26 03:31
#13
Re: Standardized item data format?
Ok, I found an HTML version of Core book that was structured enough I could scrape it for data. I currently have a *lot* of the information from Core uploaded:
http://www.firewall-darkcast.com/wiki/category/core
Still to go - Sleights, Backgrounds and Factions. And probably tons of interface issues to make sure the wiki interface works together. Once this thing is in place, I'll start adding ways to pull the information down in various formats.
That is hot to death. I've been AFK for a week, sorry. How should I start helping to fill in the other info?
@Kindalas : Awesome! Thanks a ton for sharing! I was looking at the character sheet and don't know how/where the data is stored for all of the drop downs and things. Can/May I scrape a set cell range for certain data? How would I find that?
Wed, 2011-10-26 08:56
#14
Re: Standardized item data format?
If you can find a HTML copy of Sunward and the other books that'll go a long way towards getting this put together. What I'm doing is using the HTML tags to look for regular patterns around the text that we want, so I can with enough copy/replace create a CSV out of the data. It's tricky, and takes some time to do but much less time than entering data directly. If you want to start that process on some of the other books and pass the CSV file over then I can load that data in.
Also any feedback you have on making the wiki easier to navigate on DC would be appreciated. It's one of the underused sections of that site and I know a lot of that has to do with the interface. Since the wiki is also likely to be the basis for API data pulls like you're talking about, it's worth getting the navigation right so app writers can use it.
Tue, 2011-11-01 21:34
#19
Re: Standardized item data format?
I'd add firing type, SA, SS, BF, FA, and a toggle to say if concentrated fire was possible.
And I would call the core book just EP instead of CO.
And PO for Panopticon.
It's what I've been using in the excel sheet and what I think people are used to.
Wed, 2011-11-02 13:34
#20
Re: Standardized item data format?
I'd add firing type, SA, SS, BF, FA, and a toggle to say if concentrated fire was possible.
I was talking more about the general metadata that could be applied to any and all entries, eg traits, weapons, morphs, or even skills. There would be item specific data for each type or category, but I think it would be a good idea to have a metadata node/table/whatever that always had the same fields.
And I would call the core book just EP instead of CO.And PO for Panopticon. It's what I've been using in the excel sheet and what I think people are used to.
Oh yes, agreed. Let's do that... OP was a typo 
Sun, 2011-11-27 14:14
#23
Re: Standardized item data format?
Added Data_Traits.csv
http://itsmax.net/pub/EclipsePhase/data/
It's got all traits from EP, SW, PO, and GC... I'm pretty sure. Another set of eyes on it would be helpful. If there are any columns that would be helpful, let me know. I'd considered adding an allowed or restricted morph list, but that seemed very... tedious.
Excellent work, I'm going to stat incorporating some of the data into the character generator.
For the allow morph column I'd consider having am all/bio/synth catagory so that 0 is all, 1 is bio and 2 is synth only.
And then have a 2nd column where specific morphs are enabled.
Fri, 2011-12-02 14:47
#28
Re: Standardized item data format?
Updated the morph data to have extra fields I'd left out... now includes traits and size modifier for attackers.
Also put in a numeric morphtype column that is in line with the version I did in the traits data for better cross compatability stuff...
1 = biomorphs
2 = synthmorphs
3 = pods
4 = uplifts
@Kindalas might be interested in the updated descriptions and summaries for all morphs, including GC and PO...
http://itsmax.net/pub/EclipsePhase/data/Data_Morphs.csv
And added factions.
http://itsmax.net/pub/EclipsePhase/data/Data_Factions.csv
You know at 3am well today I was thinking I wonder if it would be all right to ask thelabmonkey to write some more data fields that I'm missing.
And then I come online today and it is all done.
Thanks for being psychic
Fri, 2011-12-02 22:49
#29
Re: Standardized item data format?
Thanks for being psychic
Does "Thought Browse" count?
The least I can do after all the work you've put into the character sheet.
Sun, 2011-12-25 17:31
#31
Re: Standardized item data format?
I would be willing to try and make a parser/displayer if someone could give me JSONs to work with. :/
I have seriously considered doing something with Darkcast to store exactly this sort of information and output it in JSon or the like. I don't know if anyone would really use it though so I haven't pushed to set it up. If enough people are interested I'm sure I could set up some forms to gather the information and we can hit it up crowdsource style.
I fix broken things. Should you need to get a feature on eclipsephase.com fixed, mention it on the suggestions board and we'll look into it for you. I also fix Darkcast and the Darkcast Wiki