Partner Export/Import with Petra 2.x .ext files

pokorra

This is a summary of what is currently working:
.ext files are text files, that the old Petra 2.x has written and parsed, for transferring records of personnel or conference participants between offices.

Import partner data from an .ext file:
csharp\ICT\Petra\Server\lib\MPartner\web\ImportExt.cs, class TPartnerFileImport
method: public PartnerImportExportTDS ImportAllData(string[] ALinesToImport,
string ALimitToOption,
bool ADoNotOverwrite,
out TVerificationResultCollection AResultList);

This method gets the data of the file in a string array, and puts all partner info, that should be saved, into a typed dataset PartnerImportExportTDS.
The caller has to store the data. (eg. PartnerImportExportTDSAccess.SubmitChanges(MainDS, out VerificationResult)))

There is a web connector for that function, in csharp\ICT\Petra\Server\lib\MPartner\web\ImportExport.cs, TImportExportWebConnector,
method public static PartnerImportExportTDS ImportFromPartnerExtract(string[] ATextFileLines, out TVerificationResultCollection AVerificationResult)
The WebConnector is called already from the client, in the screen PartnerImport (csharp\ICT\Petra\Client\MPartner\Gui\PartnerImport.ManualCode.cs). That screen is comparable with the CSV import screen in Petra 2.x, where the user can determine which partners to import and which to ignore, and how to match them to existing partners. For the compatibility with Petra 2.x, it is not required to support .ext file here.

see also the test code in csharp\ICT\Testing\lib\MPartner\server\PartnerExports\PartnerExtracts.cs

This method is already used by the web client:
csharp\ICT\Petra\Server\lib\MConference\ConferenceApplications.cs,
public static bool UploadPetraExtract(string APartnerKeyFile, string AEventCode, out TVerificationResultCollection AVerificationResult)

TODO: find the TODO comments in the file csharp\ICT\Petra\Server\lib\MPartner\web\ImportExt.cs, eg. missing support for tables PmDocumentType, PmDocumentCategory, etc
TODO: see the notes to the bug sourceforge.net/apps/mantisbt/o ... php?id=133.
important: Still TODO: overwrite things in the database. I have not tested that yet (parameter ADoNotOverwrite). At the moment, only new records are added, existing records are not changed. Should show the warnings for new data, which is just logged at the moment. Petra 2.x did show a report.
Also TODO: optionally create an extract, as Petra 2.x did

Export partner data and write to .ext file:
csharp\ICT\Petra\Server\lib\MPartner\web\ExportExt.cs, class TPartnerFileExport, Method string ExportAllData(PartnerImportExportTDS AMainDS, Int32 ASiteKey, Int32 ALocationKey, StringCollection ASpecificBuildingInfo)

This seems not to be called from anywhere in OpenPetra yet.
The data export looks to be fine.
TODO: add a method to the webconnector TImportExportWebConnector in csharp\ICT\Petra\Server\lib\MPartner\web\ImportExport.cs, and run nant generateGlue
TODO: add functionality to the client, to store the returned string into a file

tim_ingham

I've been working through various issues associated with Export / Import (mostly just finding out how it works, and doing a couple of minor tweaks) and now I've come up to a more serious problem (in import).

The result of the problem is that if a user attempts to import a partner that already exists, the import crashes.

What's happening is, when a partner of a particular type is imported (in this case a FAMILY), a new FamilyRow is created, and this row is given to AddOrModifyRecord. If the record is added, or modified, then the new FamilyRow is added to the PFamily table, and otherwise it is just lost, IN WHICH CASE THERE'S A NEW PPARTNER RECORD WITH NO CORRESPONDING ENTRY IN PFAMILY. Later in the import process the code assumes that a corresponding FamilyRow will exist in PFamily, and it crashes out.

The same problem applies to Person, Church, Unit, Organisation etc.

I'm not sure what the intended behaviour is here. The AddOrModifyRecord code effectively checks whether an identical record already exists, and if so it ends up doing nothing at all (except logging). There's no return value to say that action was taken or not taken.

My guess is that there two ways to go: either AddOrModifyRecord should ALWAYS ADD the new record in the dataset, even if it's the same as existing record (Wolfgang says this is what Petra 2.x would do), or it should return false where no action was taken, then the caller can cancel the import of that record. I think this is better but I have not yet checked whether the import might fail in this situation, because where a FAMILY was not imported, a PERSON that depends on the FAMILY might fail a foreign key validity test. And ditto for all the other partner classes.

Another possibility is that this can be "fixed" on the client side - the client can detect that the records read from the file are "broken", and cancel the import at that stage. It seems to me this is the wrong place to take this action, but I don't really know enough about the design intention of the import process.

What does anyone else think about this?

pokorra

Hello Tim,

we should not do any complicated checks on the client side, since I am using the same functionality from the webclient as well.
You could return a list of questions to the user and let him confirm which records to update and which to leave as they are.
We don't have a good concept for this interaction yet, that works for both fat client and browser-based client.

When you come across different behaviour of Petra 2.x and the current OpenPetra, it is best to implement it the way as Petra 2.x did.
The current implementation in OpenPetra just worked for the situation of TeenStreet, but it is not a complete solution.
Crashing out is a good thing in that case, since it is obvious that this case was not considered yet. At least no wrong data is stored.

Ok, I have had now a look at the generated method AddOrModifyRecord. It does have a parameter: ADoNotOverwrite. If that is true, the behaviour is like you described. Only logging happens, but the record will not be modified. This was sufficient for my situation for the TeenStreet registration office.
Please try what happens if you change the parameter to false. Then the record should be modified with the value from the import file.

All the best,
Timotheus

tim_ingham

The problem that I identified is that there are situations in which AddOrModifyRecord neither adds, nor modifies. In these situations the new data row is not added to the DataSet, but there's no return value to alert the caller to this fact - the caller can (and does!) continue assuming that the new row was in fact added.

I've modified it so that it returns a bool, by changing the code in inc\template\src\ORM\DataAccess.cs but I'm not completely sure this is the right place. At the top of the file it says

// auto generated with nant generateORM
// Do not modify this file manually!

but that's not right - it's used by generateORM, but not overwritten. If this file isn't the appropriate place, where should I put this change?

pokorra

You are welcome to change the template file in inc\template\src\ORM\DataAccess.cs
The warning at the top is intended for the files that are generated, because modifying them would be useless next time the file is generated.

I have not understood yet exactly what the problem is.
If the family key exists already in the database, you get a problem during importing that family again? Perhaps we should chat about this on IRC tomorrow. I will go home now.