STG Library

Posts 1–8 of 8 · Page 1 of 1

STG Library

So yeah this is another project that I've made as a supply for one of my main's (TeamWork). Since my old EGNoSQL solution wasn't that "fast" according to a memory usage aspect. This one is kind of different. Let me show you what I mean...
First of all this is used to store a huge amount of data in a compressed archive. It automatically writes the data to the archive without the need to "load" it first on memory, which results in a huge memory save. As for the read aspect every data stored in the archive has its own header, which contains its name, length and offset. Those are used to read the data from the archive, yet without the memory hassle.
Yes if you store like 100 objects in the archive they will have 100 headers, but loading small amount of data in the RAM instead of just dumping there 100 objects like 1MB big is still better. A header length is: 8 + <number of characters in the object name>. Which is considerably smaller.

Yet this does not allow you to delete the data from the archive, which is kind of a pain in the ass (programmatically talking), I will eventually add this functionality, yet if anybody is interested in doing this (or even improving what I've already done) is free to do so, since I've included the source code.

Here's an example of how to create a new stg storage:

Code:

Log.Initialize("log.txt", LogLevel.All, true); // If you wanna log what you're doing..
STGDatabase db = new STGDatabase(@"D:\test.stg");
db.Headers.AddNewHeader(db.Data.WriteData(Encoding.UTF8.GetBytes("This is a test"), "Test")); // Adds a new header. When you read it it will not display the last character.... idk why?
db.Headers.AddNewHeader(db.Data.WriteData(Encoding.UTF8.GetBytes("This is a test 2"), "Test2")); // Same ^.
db.Commit();  // Updates the archive.
Log.WriteAway(); // Flushes the log data into the text file.

As you may have seen the db.Data.WriteData( ... ) is used to insert a new object in the storage, it then returns the header of the data. In the case above we store directly the header returned in our headers collection. Source is documented, which will allow you to see what a function does when you use it in your code.

Using the same technique but to read the data:

Code:

Header h = db.Headers.FindHeader("Test2")[0];  // Finds every header corresponding to the given name. Not case sensitive.
string hhh = Encoding.UTF8.GetString(db.Data.ReadData(h));  // The string we've wrote in the storage above.

db.Data.ReadData( ... ) allows you to read a specific area of the storage which corresponds to the given header.

This time this DOES NOT include any outside links. (Even EGNoSQL didn't since I've hex edited the exe, anyways..)

Virus Scans: #1 ~ #2

AGAIN, SOURCE CODE IS INCLUDED SO YOU ARE ABLE TO WRITE WHATEVER YOU WANT AND MODIFY THE GIVEN CLASSES. CREDITS WOULD BE APPRECIATED !

Thanks to NTAuthority, I've found in the aIWNet server source the log class and I've been using it since then.

STGLib_mpgh.net.rar10 KB · 15 downloads Clean

Download & Thanks

Content approved.

Nice documentation.

I'd recommend you use bit buffers for header declaration for even better memory efficiency.

Originally Posted by Hitokiri~

Nice documentation.

I'd recommend you use bit buffers for header declaration for even better memory efficiency.

Could you please be a little more specific ?

Originally Posted by **Seals**

Could you please be a little more specific ?

Since I haven't looked deeply into your code, excuse me if you already did something like this.

Code:

// This is the start of any database file
struct tag_DBHead{
	// Total headers in the lookup table
	// Use short because it's highly unlikely someone is ever going to use > 65536 entries
	// Optionally, use a define for it. ( Large/Small tables )
	uint16_t totalEntries;
	// The rva pointer to the start of the file table
	uint16_t ptrFileTable;
};

struct tag_EntryHead{
	// Use hashes when defining entry names since it's unlikely duplication will occur
	uint32_t hash;
	// Encoded buffer defining type used -- 3 bits
	// bit 00 - bool
	// bit 01 - int
	// bit 10 - float
	// bit 100 - double
	// bit 101 - string
	// bit 110 - char
	// bit 111 - raw blob
	uchar_t encodeType : 3;
	// Next 5 bits would then carry on to the next tag_EntryHead structure or based on the database
	// type, describe the length of the variable ( In the case of a string, blob etc. )
};

/*
	So basically, your file ends up looking like this:
	
	+0x0000 { tag_DBHead } -----------------> +0x???? { rawTableBuffer[] }
	+0x0004 { tag_EntryHead[] } ---------------------------^ ( Descriptor + Length )
*/

Originally Posted by Hitokiri~

Since I haven't looked deeply into your code, excuse me if you already did something like this.

Code:

// This is the start of any database file
struct tag_DBHead{
	// Total headers in the lookup table
	// Use short because it's highly unlikely someone is ever going to use > 65536 entries
	// Optionally, use a define for it. ( Large/Small tables )
	uint16_t totalEntries;
	// The rva pointer to the start of the file table
	uint16_t ptrFileTable;
};

struct tag_EntryHead{
	// Use hashes when defining entry names since it's unlikely duplication will occur
	uint32_t hash;
	// Encoded buffer defining type used -- 3 bits
	// bit 00 - bool
	// bit 01 - int
	// bit 10 - float
	// bit 100 - double
	// bit 101 - string
	// bit 110 - char
	// bit 111 - raw blob
	uchar_t encodeType : 3;
	// Next 5 bits would then carry on to the next tag_EntryHead structure or based on the database
	// type, describe the length of the variable ( In the case of a string, blob etc. )
};

/*
	So basically, your file ends up looking like this:
	
	+0x0000 { tag_DBHead } -----------------> +0x???? { rawTableBuffer[] }
	+0x0004 { tag_EntryHead[] } ---------------------------^ ( Descriptor + Length )
*/

Oh yeah so we pretty much first of all declare the content length of the headers then use hashes to store the name of the object and bits to define the type. That's cool but excuse me if I might have misunderstood your work but I did not see any variable there which points to the object in the storage file. Again I just took a quick look at it and might have missed out something.

You can avoid using another 32 bits per entry and manually calculate the pointer yourself by simply calculating:

index + itemTypeSize + fileBase

Unfortunately, this method would require you to load all the head entries into memory first to calculate the contiguous file size.

Originally Posted by Hitokiri~

That's like a "compressed" version of what I've done. Anyways I was thinking about encrypting the header for further security reasons. I've also considered to encrypt the whole package but it would need to store the whole thing into the memory unless I d/encrypt it byte per byte. Which is something I'm more than interested to try out...