WO2008065342A1

WO2008065342A1 - Data maps

Info

Publication number: WO2008065342A1
Application number: PCT/GB2007/004424
Authority: WO
Inventors: David Irvine
Original assignee: David Irvine
Priority date: 2006-12-01
Filing date: 2007-11-21
Publication date: 2008-06-05

Abstract

This present invention details a mechanism that allows data that has preferably been split into chunks, to be stored, managed and accessed in a way that guarantees its convenient and secure availability to its owner(s). This technique involves creating a database or 'map' of associated file chunks and their identifiers. This invention when used in conjunction with a self encryption process, allows large amounts of data to be represented in a small storage space and depicted in an extremely condensed and familiarly organised format, thereby making operations on very large data sets simple and requiring little or no training beforehand. The data management impact of these data sets is therefore enhanced in terms of the minimal transmittion bandwidth required and the type of devices the data can be stored on; i.e. PDAs, mobile phones and other portable storage devices.

Description

PATENT SPECIFICATION

TITLE: Data Maps

STATEMENT OF INVENTION:

This invention provides the ability to store file identifiers in a secure manner to allow the rebuilding of whole data elements that have been broken into parts and obfuscated and possibly encrypted. Data stored securely today, generally has a password associated with it and these passwords can give access to all data protected by them or encrypted with them. This invention provides a data map which is made up of multiple keys for each file or data element. This also means that the location of the data and its file identifiers are maintained secretly, as being the parts required to build the file. This invention identifies a mechanism for rebuilding whole data from identifiable chunks.

Another issue with systems today is that to access data the user requires to have either the data available locally or on a shared storage resource. Either way requires a storage module where the files are located, generally intact and centrally located.

This invention allows data to have multiple locations, revisions and encryption or other obfuscation techniques and for the pointer to the data to be a very small file containing the basic information to reconstitute a complete data element at any time from any location on the network. This allows the identification of which chunks make up which files. These data maps themselves preferably become discreet data chunks on the network, just like any other associated data element and are therefore undetectable as data maps.

BACKGROUND:

(perhaps a bit of a pre-amble here? _****\

US6859812 uses a system and method for differentiating private and shared files, where clustered computers share a common storage resource, Network-Attached Storage (NAS) and Storage Area Network (SAN).

US5313646 has a system which provides a copy-on-write feature which protects the integrity of the shared files by automatically copying a shared file into user's private layer when the user attempts to modify a shared file in a back layer.

WO02095545 discloses a system using a server for private file sharing which is not anonymous.

Summary of Invention

The main embodiments of this invention are as follows:

A system of data mapping which has the functional elements of:

1. Identify Chunks 2. Revision Control 3. Identify Data with Very Small File ... with the additionally linked functional elements of:

1. Duplicate Removal 2. Storage and Retrieval 3. Chunking 4. Storing Files 5. Create Map of Maps

A system of data mapping for data stored or accessed on distributed or peer to peer network,

A system of data mapping for data stored or accessed on distributed or peer to peer network is made up of inter linkage and combination of all or some of the following elements:

a. Identify data with small files b. Revision control c. Identify chucks

a. Identify data with small files i) Create maps of map b. Revision control i) Storage and retrieval c. Identify chucks i) Storing files ii) Chunking iii) Storage and retrieval iv) Duplicate removal A product for data mapping for data stored or accessed on distributed or peer to peer network is made up of inter linkage and combination of all or some of the following elements:

a. Identify data with small files b. Revision control c. Identify chucks

A product for data mapping for data stored or accessed on distributed or peer to peer network is made up of inter linkage and combination of all or some of the following elements:

a. Identify data with small files i) Create maps of map b. Revision control i) Storage and retrieval c. Identify chucks i) Storing files ii) Chunking iii) Storage and retrieval iv) Duplicate removal

A data mapping product for data stored or accessed on distributed or peer to peer network

A method of above system and product of data mapping for data stored or accessed on distributed or peer to peer network

A method of above where it is to identify data elements using a data map with only a sequence of content hashes for each chunk of data before and after encryption. 91 A method of above where each new iteration of a data element is

92 appended to the data map to create a strong revision control system.

93 A method of above where data elements are obfuscated by encryption

94 or other obfuscation technique, or similar, can be reconstructed in

95 conjunction with the data map.

96 A method of storing and retrieving these maps on an insecure network.

97 A method above where the maps can be stored in private or public

98 locations and/or biometrically accessed.

99 A system which allows data to have multiple locations, revisions and

100 encryption or other obfuscation techniques and for the pointer to the

101 data to be a very small file containing the basic information to

102 reconstitute a complete data element at any time from any location on

103 the network.

104 A system which allows the identification of which chunks to make up

105 which files.

106 A system which allows data maps which preferably become discreet

107 data chunks on the network, just like any other associated data element

108 and are therefore undetectable as data maps.

DESCRIPTION

Detailed Description:

109 (References to IDs used in descriptions of the system's functionality)

110 MID - this is the base ID and is mainly used to store and forget files.

111 Each of these operations will require a signed request. Restoring may

112 simply require a request with an ID attached.

113 PMID - This is the proxy mid which is used to manage the receiving of

114 instructions to the node from any network node such as get/ put / forget

115 etc. This is a key pair which is stored on the node - if stolen the key pair

116 can be regenerated simply disabling the thief s stolen PMID - although

117 there's not much can be done with a PMID key pair.

118 CID - Chunk Identifier, this is simply the chunkid.KID message on the

119 net.

120 TMID - This is today's ID a one time ID as opposed to a one time

121 password. This is to further disguise users and also ensure that their MID

122 stays as secret as possible.

123 MPID - The maidsafe.net public ID. This is the ID to which users can add

124 their own name and actual data if required. This is the ID for messenger,

125 sharing, non anonymous voting and any other method that requires we

126 know the user.

127 MAID - this is basically the hash of and actual public key of the MID. this

128 ID is used to identify the user actions such as put / forget / get on the

129 maidsafe.net network. This allows a distributed PKI infrastructure to exist

130 and be automatically checked. 131 KID - Kademlia ID this can be randomly generated or derived from

132 known and preferably anonymous information such as an anonymous

133 public key hash as with the MAID.. In this case we use kademlia as the

134 example overlay network although this can be almost any network

135 environment at all.

136 MSID - maidsafe.net Share ID, an ID and key pair specifically created for

137 each share to allow users to interact with shares using a unique key not

138 related to their MID which should always be anonymous and separate.

Linked elements for Data Maps (Figure 1)

139 The Data Maps invention consists of 3 key functional elements, with a

140 further 5 functional elements being linked with.

141 The key functional elements are:

142 P9 - Identify Chunks

143 P10 - Revision Control

144 P11 - Identify Data with Very Small File

145 The linked functional elements are:

146 P5 - Duplicate Removal

147 P7 - Chunking

148 P6 - Storing Files

149 P4 - Storage and Retrieval

150 P15 - Create Map of Maps

151 The data maps (PT3) itself is made up from linkage of elements, identify

152 chunks (P9), preferably revision control (P10), and identify data with

153 small files (P11) to provide a data mapping system for data location or

154 naming convention storing or accessing on distributed or peer to peer

155 network. In addition, identify chunks element (P9) is dependent on sub- 156 elements storage and retrieval (P4) and preferably sub-elements

157 chunking (P7) and preferably generate sub-elements duplicate removal

158 (P5) and sub-elements storing files (P6), revision control (P 10) generates

159 sub-element storage and retrieval (P4), and preferably identify data with

160 small data elements (P11) generate sub-element create map of maps

161 (P15) to allow private data sharing.

Storing Initial Chunk (Figure 2)

162 According to a related aspect of this invention, a file is chunked or

163 split into constituent parts (1) this process involves calculating the chunk

164 size, preferably from known data such as the first few bytes of the hash

165 of the file itself and preferably using a modulo division technique to

166 resolve a figure between the optimum minimum and optimum maximum

167 chunk sizes for network transmission and storage.

168 Preferably each chunk is then encrypted and obfuscated in some manner

169 to protect the data. Preferably a search of the network is carried out

170 looking for values relating to the content hash of each of the chunks (2).

171 If this is found (4) then the other chunks are identified too, failure to

172 identify ail chunks may mean there is a collision on the network of file

173 names or some other machine is in the process of backing up the same

174 file. A back-off time is calculated to check again for the other chunks. If

175 all chunks are on the network the file is considered backed up and the

176 user will add their MID signature to the file after preferably a challenge

177 response to ensure there a valid user and have enough resources to do

178 this.

179 If no chunks are on the net the user preferably via another node (3) will

180 request the saving of the first copy (preferably in distinct time zones or

181 other geographically dispersing method). 182 The chunk will be stored (5) on a storage node allowing us to see the

183 PMID of the storing node and store this.

184 Then preferably a Key.value pair of chunkid. public key of initiator is

185 written to net creating a Chunk ID (CID) (6)

Storage and Retrieval (Figure 1 - P4)

186 According to a related aspect of this invention, the data is stored in

187 multiple locations. Each location stores the locations of its peers that hold

188 identical chunks (at least identical in content) and they all communicate

189 regularly to ascertain the health of the data. The preferable method is as

190 follows:

191 Preferably the data is copied to at least three disparate locations.

192 Preferably each copy is performed via many nodes to mask the initiator.

193 Preferably each local copy is checked for validity and checks are made

194 that the preferably other 2 copies are also still valid.

195 Preferably any single node failure initiates a replacement copy being

196 made in another disparate location and the other associated copies are

197 updated to reflect this change.

198 Preferably the steps of storing and retrieving are carried out via other

199 network nodes to mask the initiator.

200 Preferably, the method further comprises the step of renaming all files

201 with a hash of their contents. 202 Preferably each chunk may alter its name by a known process such as a

203 binary shift left of a section of the data. This allows the same content to

204 exist but also allows the chunks to appear as three different bits of data

205 for the sake of not colliding on the network.

206 Preferably each chunk has a counter attached to it that allows the

207 network to understand easily just how many users are attached to the

208 chunk - either by sharing or otherwise. A user requesting a 'chunk forget'

209 will initiate a system question if they are the only user using the chunk

210 and if so the chunk will be deleted and the user's required disk space

211 reduced accordingly. This allows users to remove files no longer required

212 and free up local disk space. Any file also being shared is preferably

213 removed from the user's quota and the user's database record or data

214 map (see later) is deleted.

215 Preferably this counter is digitally signed by each node sharing the data

216 and therefore will require a signed 'forget' or 'delete' command.

217 Preferably even 'store', 'put', 'retrieve' and 'get' commands should also

218 be either digitally signed or preferably go through a PKI challenge

219 response mechanism.

220 To ensure fairness preferably this method will be monitored by a

221 supemode or similar to ensure the user has not simply copied the data

222 map for later use without giving up the disk space for it. Therefore the

223 user's private ID public key will be used to request the forget chunk

224 statement. This will be used to indicate the user's acceptance of the

225 'chunk forget' command and allow the user to recover the disk space.

226 Any requests against the chunk will preferably be signed with this key

227 and consequently rejected unless the user's system gives up the space

228 required to access this file. 229 Preferably each user storing a chunk will append their signed request to

230 the end of the chunk in an identifiable manner i.e. prefixed with 80 - or

231 similar.

232 Forgetting the chunk means the signature is removed from the file. This

233 again is done via a signed request from the storage node as with the

234 original backup request.

235 Preferably this signed request is another small chunk stored at the same

236 location as the data chunk with an appended postfix to the chunk

237 identifier to show a private ID is storing this chunk. Any attempt by

238 somebody else to download the file is rejected unless they first subscribe

239 to it, i.e. a chunk is called 12345 so a file is saved called 12345 <signed

240 store request>. This will allow files to be forgotten when all signatories to

241 the chunk are gone. A user will send a signed 'no store' or 'forget' and

242 their ID chunk will be removed, and in addition if they are the last user

243 storing that chunk, the chunk is removed. Preferably this will allow a

244 private anonymous message to be sent upon chunk failure or damage

245 allowing a proactive approach to maintaining clean data.

246 Preferably as a node fails the other nodes can preferably send a

247 message to all sharers of the chunk to identify the new location of the

248 replacement chunk.

249 Preferably any node attaching to a file then downloading immediately

250 should be considered an alert and the system may take steps to slow

251 down this node's activity or even halt it to protect data theft.

Identify Chunks (Figure 1 - P9)

252 According to a related aspect of this invention, a chunk's original

253 hash or other calculable unique identifier will be stored. This will be 254 stored with preferably the final chunk name. This aspect defines that

255 each file will have a separate map preferably a file or database entry to

256 identify the file and the name of its constituent parts. Preferably this will

257 include local information to users such as original location and rights

258 (such as a read only system etc.). Preferably some of this information

259 can be considered shareable with others such as filename, content hash

260 and chunks names.

ID Data with Small File (Figure 1 - P11)

261 According to a related aspect of this invention; these data maps may

262 be very small in relation to the original data itself allowing transmission of

263 files across networks such as the internet with extreme simplicity,

264 security and bandwidth efficiency. Preferably the transmission of maps

265 will be carried out in a very secure manner, but failure to do this is akin to

266 currently emailing a file in its entirety.

267 This allows a very small file such as the data map or database record to

268 be shared or maintained by a user in a location not normally large

269 enough to fit a file system of any great size, such as on a PDA or mobile

270 phone. The identification of the chunk names, original names and final

271 names are all that is required to retrieve the chunks and rebuild the file

272 with certainty.

273 With data maps in place a user's whole machine, or all its data, can exist

274 elsewhere. Simply retrieving the data maps of all data, is all that is

275 required to allow the user to have complete visibility and access to all

276 their data as well as any shared files they have agreed to. Revision Control (Figure 1 - P10)

277 According to a related aspect of this invention, as data is updated

278 and the map contents alter to reflect the new contents, this will preferably

279 not require the deletion or removal of existing chunks, but instead allow

280 the existing chunks to remain and the map appended to with an

281 indication of a new revision existing. Preferably further access to the file

282 will automatically open the last revision unless requested to open an

283 earlier revision.

284 Preferably revisions of any file can be forgotten or deleted (preferably

285 after checking the file counter or access list of sharers as above). This

286 will allow users to recover space from no longer required revisions.

Claims

287

288 1. A system of data mapping for data stored or accessed on distributed or

289 peer to peer network is which allows creation of a database or 'map' of

290 associated file chunks and their identifiers and provides a mechanism

291 that allows data that has preferably been split into chunks, to be stored,

292 managed and accessed in a way that guarantees its convenient and

293 secure availability to its owner(s), this system comprises of following

294 steps:

295 a. Identify data with small files

296 b. Revision control

297 c. Identify chunks

298 the above combination provides a unique system with cumulative and

299 synergistic benefits to allow people to effectively control access to their

300 own digital resources. 301

302 2. A system of claimi of data mapping for data stored or accessed on

303 distributed or peer to peer network is which allows creation of a database

304 or 'map' of associated file chunks and their identifiers and provides a

305 mechanism that allows data that has preferably been split into chunks, to

306 be stored, managed and accessed in a way that guarantees its

307 convenient and secure availability to its owner(s), this system comprises

308 of following steps:

309 a. Identify data with small files, which further comprises of create maps

310 of map,

311 b. Revision control, which further comprises ofstorage and retrieval

312 c. Identify chunks, which further comprises of storing files, chunking,

313 storage & retrieval and duplicate removal

314 the above combination provides a unique system with cumulative and

315 synergistic benefits to allow people to effectively control access to their

316 own digital resources. 317 il8 3. A product of data mapping for data stored or accessed on distributed or

J 19 peer to peer network is which allows creation of a database or 'map' of

520 associated file chunks and their identifiers and provides a mechanism

321 that allows data that has preferably been split into chunks, to be stored,

322 managed and accessed in a way that guarantees its convenient and

323 secure availability to its owner(s), this product comprises of following

324 steps:

325 a. Identify data with small files

326 b. Revision control

327 c. Identify chunks

328 the above combination provides a unique product with cumulative and

329 synergistic benefits to allow people to effectively control access to their

330 own digital resources. 331

332 4. A product of claim 3 of data mapping for data stored or accessed on

333 distributed or peer to peer network is which allows creation of a database

334 or 'map' of associated file chunks and their identifiers and provides a

335 mechanism that allows data that has preferably been split into chunks, to

336 be stored, managed and accessed in a way that guarantees its

337 convenient and secure availability to its owner(s), this product comprises

338 of following steps:

339 a. Identify data with small files, which further comprises of create maps

340 of map,

341 b. Revision control, which further comprises ofstorage and retrieval

342 c. Identify chunks, which further comprises of storing files, chunking,

343 storage & retrieval and duplicate removal

344 the above combination provides a unique product with cumulative and

345 synergistic benefits to allow people to effectively control access to their

346 own digital resources. 347

348 5. A method of claim 1-4 where it is to identify data elements using a data

349 map with only a sequence of content hashes for each chunk of data

350 before and after encryption; 351

352 6. A method of claims 1-4 of storing and retrieving these maps on an

353 insecure network; 354

355 7. A method of claim 6, where each, new iteration of a data element is

356 appended to the data map to create a strong revision control system; 357

358 8. A method of claim 6, where data elements are obfuscated by encryption

359 or other obfuscation technique, or similar, can be reconstructed in

360 conjunction with the data map; 361

362 9. A method of claim 8, where the maps can be stored in private or public

363 locations and/or biometrically accessed; 364

365 10. A system of claims 1-2 which allows data to have multiple locations,

366 revisions and encryption or other obfuscation techniques and for the

367 pointer to the data to be a very small file containing the basic information

368 to reconstitute a complete data element at any time from any location on

369 the network; 370

371 11. A system of claims 1 -2 which allows the identification of which chunks to

372 make up which files; 373

374 12. A system of claims 1-2 which allows data maps which preferably become

375 discreet data chunks on the network, just like any other associated data

376 element and are therefore undetectable as data maps;