NPK: Process1D.py

NPK: Process1D.py Wed Oct 14 15:25:36 2009

    This module contains all the routines needed to process 1D NMR spectra
    
    The following functions realize a set of operations, needed for NMR 1D processing,
    You will find operation for FT analysis, MaxEnt Analysis,
    Inverse Fourier, etc..
    
    Processing are divided in pre - FT - Post phases.
    For instance a typical 1D procesing, would be

    pre()
    ft()
    post()

    but other combinations are possible.

    Most functions take the following arguments :
    arguments :
    audit
        the opened audit file, if empty or set to none, audit will go to the stdout
    filein      the file name of the input file, 
        will loaded before operation, if name is "memory", then processing takes place on the current 2D kernel buffer
    fileout     the file name of the input file, can be "memory"
        will written after operation, if name is "memory", then processing results is left in 2D kernel buffer
    p_in        the dictionary containing all the processing parameters
        most entries are optionnal as entries are protected with try / except
    f_in        the dictionary containing all the details on the spectrum "filein"
    f_out       details on the spectrum "fileout" will be put in this dictionary
    location    the directory where the output files will be written

FT1D(audit,p_in_arg,f_in,f_out,inputfilename,outputfilename)
		FT processing of a 1D FID based on pre_ft_1d() ft_1d() post_ft_1d()

MaxEnt1D(audit,p_in_arg,f_in,f_out,inputfilename,outputfilename)
		MaxEnt processing of a 1D FID based on pre_ft_1d() maxent_1d() post_maxent_1d()

pre_ft_1d( audit, filein, fileout, p_in, f_in, f_out)

This macro realizes the pre FT operation on a 1D FID

- fid_noise        : evaluate of noise and offset levels in FID
- dc_offset        : corrects for constant offset in FID
- causalize        : changes DSP processed FID (Bruker) to causal FID's by Hilbert transform
- flatten_solvent  : removes solvent signal by FID analysis
- left_shift       : drops first points of the FID 
- right_shift      : adds empty points on the beginning of the FID
- back_extend      : reconstructs missing points in the beginning of the FID by LP analysis

Input Domain in F1:   time
Output Domain in F1:   time
Dimensionality:   1

Author:   Marc-Andre Delsuc
Version:   6.0

Actions

fid_noise: evaluate noise and offset, estimated from the last points
fid_noise_zone:   float / default 0.2
zone of the FID (expressed in % from the end) where to compute noise
conditions:   [ (fid_noise_zone > 0.0) and (fid_noise_zone<=1)]
returned value:   fid_offset
constant offset of the FID, estimated on the last points of the dataset (defined by fid_noise_zone)
returned value:   NoiseInTimeDomain
estimate of the fid noise, from the standard deviation of the last points of the dataset (defined by fid_noise_zone)

dc_offset: removes a constant level in the fid, estimated from the last points
default has been set to off (0), as it seems the
dc_offset: boolean / default 0

causalize: removes the non causal header found in certain (Bruker) implementation of the digital filter.
causalize:   boolean / default 0
input value:   zerotimeposition
returned value:   causalize
contains the correction applied by causalize

flatten_solvent: removes water from fit of the FID in the time domain
flatten_solvent: boolean / default 0
flat_solv_mode: enum polynomial moving_average polynomial+moving_average moving_average+polynomial / default polynomial
algorithm used

left_shift: shifts the FID to the left by dropping data points
left_shift:   boolean / default 0
left_shift_size:   integer / default 2
number of points to shift
conditions:   [ (left_shift_size > 0) and (left_shift_size < get_si1_1d()) ]

right_shift: shifts the FID to the right by adding zero data points
right_shift:   boolean / default 0
right_shift_size:   integer / default 2
number of points to shift
conditions:   [ (right_shift_size > 0) ]

b_extend: extends the FID backward for reconstruction of initial missing points
b_extend:   boolean / default 0
b_extend_size:   integer / default 2
number of missing points to reconstruct
conditions:   [ b_extend_size > 0 ]
b_extend_algo:   enum burg svd / default burg
algorithm used
returned value:   b_extend_order
the order used during back_extend
returned value:   b_extend_size
the number of points added by back_extend

ft_1d( audit, filein, fileout, p_in, f_in, f_out )

This macro realizes the FT operation on a 1D FID

- truncate      : truncates the FID by removing the last points
- lp_extend     : extend FID with a Linear Prediction algorithm
- apodize       : standard apodisation by a window
- fourier_transform : performs the Fourier transform
- causal_corr   : performs causal correction of the spectrum if not done on the FID
- reverse       : reverses the spectral axis after FT


Input Domain in F1:   time
Output Domain in F1:   frequency
Dimensionality:   1

Author:   Marc-Andre Delsuc
Version:   6.0

Actions

truncate: truncates the FID by removing the last points
truncate:   boolean / default 0
trunc_size:   integer / default get_si1_1d()
FID size after truncation
conditions:   [ (trunc_size > 0 ) and (trunc_size <= get_si1_1d() ) ]

lp_extend: extend FID with a Linear Prediction algorithm
lp_extend:   boolean / default 0
lp_ext_size:   integer / default 2*get_si1_1d()
final size of FID
conditions:   [ lp_ext_size > get_si1_1d() ]
lp_ext_algo:   enum burg mirror lpsvd lpsvd_stable / default burg
algorithm used
burg and mirror are much faster, svd_stable is much slower
mirror is to be used when the phase of the spectrum is known before hand (see lp_ext_off)
lp_ext_off:   integer / default 0
offset determines the position of the t=0 point, used by mirror algo
0 no shift : in-phase data set.
-1 acquisition started exactly half a dwell after t=0 - (will need phase 0 180)
lp_ext_off>0 acquisition started exactly n dwell before t=0
conditions:   [ lp_ext_off > -1 ]
lp_ext_order:   integer / default (min(get_si1_1d()/2,20))
the size of the prediction polynomial used for LP, a rough estimate of the complexity
conditions:   [ (lp_ext_order < get_si1_1d()/2 ) and (lp_ext_order > 2) ]
lp_ext_apod:   boolean / default 1
apply a sine bell apodisation after LP extenstion.

apodize: standard apodisation by a window
apodize: boolean / default 1
apodisation: string / default "expbroad(0.5)"
the string describing the apodisation to apply
"apodisation" should be a suite of window apodisation commands
ie : 'sin(0.5)' or 'exbroad(0.2), sin(0.5)'
the following predefined functions are implemnted
sin (sine bell) sqsin (squared sine bell) expbroad (exponential)
gaussbroad (gaussian) gaussenh (gaussian enhancement)

fourier_transform: performs the Fourier transform
size after FT is determined automatically from FID size and apodisation.
the optimum size is determined heuristically from
- the size of the FID (after truncation) : default is to zerofill once
- the apodisation : resolution is not more than twice the filtered linewidth
fourier_transform:   boolean / default 1
ask_for_ftsize:   boolean / default 0
if set, the final size is determined by ft_size, automatically determined otherwise
it will be extended to the next 2^n
ft_size:   integer / default 2*power2(get_si1_1d())
ft_type:   enum none ft_seq ft_sim ft rft / default ft_sim
the Fourier transform algorithm, depends on spectrometer and acquisition scheme
conditions:   [ ispower2(ft_size) ]
returned value:   size_for_ft
contains the data size right after FT (will usually be different from final size)

causal_corr: performs causal correction of the spectrum if not done on the FID
causal_corr:   boolean / default 1
exclusive parameters:   causal [ causalize causal_corr ]
input value:   axisF1_zerotimeposition
returned value:   causalize
contains the correction applied by causalize

reverse: reverses the spectral axis after FT
reverse: boolean / default 0

post_ft_1d( audit, filein, fileout, p_in, f_in, f_out)

This macro realizes the Post processing of a 1D spectrum

- modulus       : takes the complex modulus of the spectrum
- phase         : applies a phase correction to the spectrum
- autophase     : automatically computes the phase correction of the spectrum
- invHilbert    : apply an inverse Hilbert transform
- calibration   : calibrate the ppm scale on the spectrum
- spectral_zone : extract one spectral zone of the spectrum
- baseline_correction : applies a baseline correction to the spectrum
- smoothing     : apply a smoothing filter to the data-set
- median        : apply a median filter to the data-set
- derivative    : compute the nth derivative of the data-set
- spec_noise    : evaluate noise, estimated by finding an empty zone



Input Domain in F1:   frequency
Output Domain in F1:   frequency
Dimensionality:   1

Author:   Marc-Andre Delsuc
Version:   6.0

Actions

modulus: takes the complex modulus of the spectrum
modulus: boolean / default 0

phase: applies a phase correction to the spectrum
phase:   boolean / default 0
phase_0:   float / default 0.0
global order phase correction
phase_1:   float / default 0.0
1st order phase correction

autophase: automatically computes the phase correction of the spectrum
autophase:   boolean / default 1
phase_algo:   enum apsl apmin / default apmin
algorithm used
returned value:   autophase_0
contains the value of the global order phase correction applied by autophase
returned value:   autophase_1
contains the value of the 1st order phase correction applied by autophase

invHilbert: apply an inverse Hilbert transform.
This operation takes a complex spectrum, and generates a real spectrum in-place
i.e. with twice the number of real points.
invHilbert: boolean / default 0

calibration: calibrate the ppm scale on the spectrum
calibration: float / default 0.0
the coordinate in ppm of the right-most point in the spectrum

spectral_zone: extract one spectral zone of the spectrum
spectral_zone:   boolean / default 0
spec_zone_left:   float / default 10.0
the left border of the extract zone, in unit
spec_zone_left_unit:   enum ppm hz index / default ppm
the unit in which spec_zone_left is given
spec_zone_right:   float / default 0.0
the right border of the extract zone, in unit
spec_zone_right_unit:   enum ppm hz index / default ppm
the unit in which spec_zone_right is given
conditions:   (spec_zone_left{index} < spec_zone_right{index})
returned value:   spec_zone_left
the left coordinate of the extracted spectral zone in index
returned value:   spec_zone_right
the right coordinate of the extracted spectral zone in index

baseline_correction: applies a baseline correction to the spectrum
baseline:   boolean / default 1
bcorr_algo:   enum offset linear spline quest polynomial moving_average / default "offset"
offset : removes a automatically determined offset
linear - spline : uses the list of user determined pivot points
to define the baseline, then fit with a straight line or a spline
quest : reconstruct the beginning of the FID using Linear Prediction technics, should usually be followed by offset
polynomial - moving_average : uses statistics to separate signal from baseline,
then apply a polynomial or moveing average correction.
in more than one term is given, the correction are applied in sequence.
spec_noise_n:   integer / default 30
used by the offset algorithm to determine the offset to correct
bcorr_pivots:   Integerlist / default "10 (get_si1_1d()-10)"
pivot points used by the linear or spline algorithm
conditions:   (bcorr_pivots > 0 and bcorr_pivots <= get_si1_1d())
bcorr_radius:   integer / default 1
radius around pivot points used by the linear or spline algorithm
bcorr_order:   integer / default 10
conditions:   (bcorr_order > 0 and bcorr_order < get_si1_1d() and bcorr_order == 2*n)
order (number of points corrected in the time domain) of the quest algorithm
returned value:   spec_offset
offset computed when using the offset algo

smoothing: apply a smoothing filter to the data-set
smoothing:   boolean / default 0
smooth_w:   integer / default 5
size of the smoothing window
smooth_iteration:   / default 1
number of loop
conditions:   [ smooth_w > 0 ]

median: apply a median filter to the data-set
median:   boolean / default 0
median_w:   integer / default 6
size of the median filtering window
median_i:   integer <= median_w / default 3
index of point to keep in the median filtering window
conditions:   [ median_w > 0 ]
conditions:   [ (median_i > 0) and (median_i <= median_w) ]

derivative: compute the nth derivative of the data-set
derivative:   boolean / default 0
deriv_nth:   integer > 0 / default 1
deriv_smooth:   integer > 0 / default 2
binomial smoothing applied before derivative

select_state: permit to choose the state complex / real of the output file
complex data are changed to real by dropping the imaginary part
real data are changed to complex by computing the Hilbert transform with tocomplex()
this is usually not required as all processing commands prepare the state themselves
select_state: boolean / default 0
actuallly does the selection
f1_state: enum ignore complex real / default ignore
force the f1 axis to real or complex. ignore will let the axis unchanged

spec_noise: evaluate noise, estimated by finding an empty zone
spec_noise_n: integer / default 30
number of different zones where noise is evaluated
returned value: spec_std_noise
estimate of the noise in the spectrum

write_file_1d( audit, fileout )
		write a 1D file and update the audittrail

pp_1d( audit, filein, filepeak, p_in, f_in, f_out )

This macro realizes the peak picking of a 1D spectrum
- spec_noise : evaluate noise, estimated by finding an empty zone
- prefilter  : smooths the spectrum before peak-picking, modification is not stored permanently
- restrict   : restricts the peak picking to a certain spectral zone 
- peakpick   : do the peak picking, by detecting local extrema 
- aggregate  : sorts peak list to aggregate peaks close from each other

    
Input Domain in F1:   frequency
Output Domain in F1:   frequency
Dimensionality:   1

Author:   Marc-Andre Delsuc
Version:   6.0

Actions

spec_noise: evaluate noise, estimated by finding an empty zone
spec_noise:   boolean / default 0
spec_noise_n:   integer / default 30
number of different zones where noise is evaluated
returned value:   NoiseInFrequencyDomain
estimate of the noise in the spectrum

prefilter: smooths the spectrum before peak-picking, modification is not stored permanently
prefilter: boolean / default 1
prefilter_value: float / default 3.0
pre-smoothing parameter in Hz

restrict: restricts the peak picking to a certain spectral zone
restrict:   boolean / default 0
restrict_left:   float / default 10.0
the left border of the restrict zone, in unit
restrict_left_unit:   enum ppm hz index / default ppm
the unit in which restrict_left is given
restrict_right:   float / default 0.0
the right border of the restrict zone, in unit
restrict_right_unit:   enum ppm hz index / default ppm
the unit in which restrict_right is given
conditions:   (restrict_left{index} < restrict_right{index})
returned value:   restrict_left
the left coordinate of the restrict zone in index
returned value:   restrict_right
the right coordinate of the restrict zone in index

peakpick: do the peak picking, by detecting local extrema
peakpick:   boolean / default 1
noise_thresh:   float / default 10.0
minimum peak/noise ratio for peak detection
ratio_thresh:   float / default 30.0
maximum tallest peak / peak ratio for peak detection
peak_sign:   enum positive negative both / default positive
sign of the peaks to detect
returned value:   nb_detect_peaks
the number of detected peaks
returned value:   low_limit
low limit for peak detection

aggregate: sorts peak list to aggregate peaks close from each other
aggregate: boolean / default 0
aggregate_value: float / default 3.0
distance parameter in Hz

ift_1d( audit, filein, filout, p_in, f_in, f_out)

This macro Computes the Inverse Fourier transform of a 1D spectrum

- apodize       : standard apodisation
- inverseFourier : performs the inverse Fourier transform
- reverse       : reverses the spectral axis after FT
    
Input Domain in F1:   frequency
Output Domain in F1:   time
Dimensionality:   1

Author:   Marc-Andre Delsuc
Version:   6.0

Actions

apodize: standard apodisation
apodize: boolean / default 0
apodisation: string / default "sin 0.5"
the string describing the apodisation to apply
see apodise.g for details

inverseFourier: performs the inverse Fourier transform
inverseFourier:   boolean / default 1
ask_for_iftsize:   boolean / default 0
if set, the final size is determined by ift_size, automatically determined otherwise
size for FT is determined by current size
it will be extended to the next 2^n
ift_size:   integer / default 2*power2(get_si1_1d())
ft_type:   enum none ift ifthilbert / default ifthilbert
two iFT algorithms, ifthilbert discard the imaginary part if present and computes it using Hilbert transform
conditions:   [ ispower2(ift_size) ]
returned value:   size_for_ift contains the data size right after iFT

reverse: reverses the spectral axis after FT
reverse: boolean / default 0

integ_1d( audit, filein, integratout, p_in, f_in, f_out)

This macro Computes integrales of a 1D spectrum
- integral    : omputes integral positions from peaks

    
Input Domain in F1:   frequency
Output Domain in F1:   frequency
Dimensionality:   1

Author:   Marc-Andre Delsuc
Version:   6.0

Actions

integral: computes integral positions from peaks
integ_aggregate:   float / default 30
peak aggregatian width, in Hz
integ_extention:   float / default 15
integral extension, in Hz
returned value:   nb_integral
number of integral zones determined

maxent_1d( audit, filein, fileout, p_in, f_in, f_out)

This macro realizes the MaxEnt analysis of a 1D FID
- freq_massage  : computes a temporary Fourier transform
- truncate   : truncates the FID by removing the last points
- preconvoluate  :apply a preconvolution before analysis, this may help stability of the algorithm and enhance noise rejection
- partialsampling   : set-up for processing data partially sampled in the time domain
- deconvoluate : apply a deconvolution during analysis,
- maxent : apply MaxEnt analysis,

    
Input Domain in F1:   frequency
Output Domain in F1:   frequency
Dimensionality:   1

Author:   Marc-Andre Delsuc
Version:   6.0

Actions

freq_massage: computes a temporary Fourier transform
freq_massage:   boolean / default 1
ft_type:   enum none ft_seq ft_sim ft rft / default ft
the Fourier transform algorithm, depends on spectrometer and acquisition scheme
phase:   boolean / default 0
phase_0:   float / default 0.0
global order phase correction
phase_1:   float / default 0.0
1st order phase correction

truncate: truncates the FID by removing the last points
truncate:   boolean / default 0
trunc_size:   integer / default get_si1_1d()
FID size after truncation
conditions:   [ (trunc_size > 0 ) and (trunc_size <= get_si1_1d() ) ]

preconvoluate: apply a preconvolution before analysis, this may help stability of the algorithm and enhance noise rejection
preconvoluate: boolean / default 0
preconvolution: string / default "expbroad(1)"
the string describing the apodisation to apply
see apodise.g for details

partialsampling: set-up for processing data partially sampled in the time domain
partialsampling: boolean / default 0
samplingfile: string / default vplist
the filename of the file containing the list of sample to use,
file is relative to the directory containg the input file
see sampling.g for details

deconvoluate: apply a deconvolution during analysis,
deconvoluate: boolean / default 1
deconvolution: string / default "expbroad(1)"
the string describing the apodisation to apply
see apodise.g for details

maxent: apply MaxEnt analysis,
maxent:   boolean / default 1
me_preset:   boolean / default 0
determines convergence parameters from a given preset or gives the details
preset_value:   enum 1 2 3 4 5 / default 3
preset for maxent; 1 is faster, 5 is more accurate
iteration:   integer / default 100
global number of iterations - set by preset
control:   integer / default 10
number of iterations between control step - set by preset
lambsp:   float / default 5.0
the value of the lambda controlling parameter LAMBSP in the Gifa algorithm
lambcont:   enum 0 1 2 / default 1
the value of the lambda controlling parameter LAMBCONT in the Gifa algorithm
me_size:   integer / default 2*get_si1_1d()
size for MaxEnt reconstruction
noise_weight:   float / default 1.0
the noise used during the MaxEnt iteration is weigthed with this scalar,
this permits to compensate for over- or under-estimate of the noise level
conditions:   [ noise_weight > 0.0 ]
conditions:   [ ispower2(me_size) ]

post_maxent_1d( audit, filein, fileout, p_in, f_in, f_out)

    post processing of a 1D spectrum processed by MaxEnt

Actions

calibration: calibrate 0ppm on the spectrum
calibration: float / default 0.0
the coordinate in ppm of the right-most point in the spectrum

spectral_zone: extract one spectral zone of the spectrum
spectral_zone:   boolean / default 0
spec_zone_left:   float / default 10.0
the left border of the extract zone, in unit
spec_zone_left_unit:   enum ppm hz index / default ppm
the unit in which spec_zone_left is given
spec_zone_right:   float / default 0.0
the right border of the extract zone, in unit
spec_zone_right_unit:   enum ppm hz index / default ppm
the unit in which spec_zone_right is given
conditions:   (spec_zone_left{index} < spec_zone_right{index})
returned value:   spec_zone_left the left coordinate of the extracted spectral zone in index
returned value:   spec_zone_right the right coordinate of the extracted spectral zone in index

baseline correction: baseline correction of the spectrum
baseline:   boolean / default 1
bcorr_algo:   enum offset linear spline polynomial moving_average polynomial+moving_average moving_average+polynomial / default linear
bcorr_pivots:   Integerlist / default 10.0
pivot points used by the linear or spline algorithm
conditions:   (index > 0 and index <= get_si1_1d())
bcorr_radius:   integer > 0 / default 3
radius around pivot points used by the linear or spline algorithm
returned value:   spec_offset
offset computed from the pivot points when using the offset algo